[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:14 dist-test-slave-4t1f krb5kdc[396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:14 dist-test-slave-4t1f krb5kdc[396](info): set up 2 sockets
Jun 19 06:27:14 dist-test-slave-4t1f krb5kdc[396](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 06:27:15.089946 389 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:15 dist-test-slave-4t1f krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:15.248783 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38729
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:15.564949 409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:15.565641 409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:15.565744 409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:15.572453 409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:15.572554 409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:15.572597 409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:15.572665 409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:15.572710 409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:15.582089 409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:15.583262 409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:15.584383 409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:15.596230 414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:15.596194 415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:15.596264 417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:15.597110 409 server_base.cc:1034] running on GCE node
I20260619 06:27:15.597914 409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:15.599241 409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:15.600451 409 hybrid_clock.cc:648] HybridClock initialized: now 1781850435600386 us; error 55 us; skew 500 ppm
I20260619 06:27:15.600800 409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:15 dist-test-slave-4t1f krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850435, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:15.603865 409 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:15.605736 409 webserver.cc:460] Webserver started at http://127.0.97.126:42049/ using document root <none> and password file <none>
I20260619 06:27:15.606389 409 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:15.606535 409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:15.606853 409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:15.609891 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "9912d28f52f245a7bc167858c500c053"
format_stamp: "Formatted at 2026-06-19 06:27:15 on dist-test-slave-4t1f"
server_key: "fe2e8aa0dfd8ed47101aec1982d5b232"
server_key_iv: "2e87cc570a35b5c0b55053f177731e2b"
server_key_version: "encryptionkey@0"
I20260619 06:27:15.610634 409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "9912d28f52f245a7bc167858c500c053"
format_stamp: "Formatted at 2026-06-19 06:27:15 on dist-test-slave-4t1f"
server_key: "fe2e8aa0dfd8ed47101aec1982d5b232"
server_key_iv: "2e87cc570a35b5c0b55053f177731e2b"
server_key_version: "encryptionkey@0"
I20260619 06:27:15.615752 409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260619 06:27:15.619024 424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:15.620275 409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:27:15.620473 409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "9912d28f52f245a7bc167858c500c053"
format_stamp: "Formatted at 2026-06-19 06:27:15 on dist-test-slave-4t1f"
server_key: "fe2e8aa0dfd8ed47101aec1982d5b232"
server_key_iv: "2e87cc570a35b5c0b55053f177731e2b"
server_key_version: "encryptionkey@0"
I20260619 06:27:15.620666 409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:15.638454 409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:15.642762 409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:15.643074 409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:15.661481 409 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38729
I20260619 06:27:15.661926 475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38729 every 8 connection(s)
I20260619 06:27:15.663206 409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:15.666486 476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:15.673278 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 409
I20260619 06:27:15.673561 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:15.675316 389 external_mini_cluster.cc:1428] Setting key d404a08af5f2c76d3a30c633a8ff9818
I20260619 06:27:15.675844 476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053: Bootstrap starting.
I20260619 06:27:15.679726 476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:15.680979 476 log.cc:824] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:15.683351 476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053: No bootstrap required, opened a new log
I20260619 06:27:15.691357 476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9912d28f52f245a7bc167858c500c053" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38729 } }
I20260619 06:27:15.691706 476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:15.691835 476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9912d28f52f245a7bc167858c500c053, State: Initialized, Role: FOLLOWER
I20260619 06:27:15.692425 476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9912d28f52f245a7bc167858c500c053" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38729 } }
I20260619 06:27:15.692641 476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:15.692772 476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:15.692945 476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:15.694988 476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9912d28f52f245a7bc167858c500c053" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38729 } }
I20260619 06:27:15.695451 476 leader_election.cc:302] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9912d28f52f245a7bc167858c500c053; no voters:
I20260619 06:27:15.696035 476 leader_election.cc:288] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:15.696161 481 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:15.698328 476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:15.702057 481 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [term 1 LEADER]: Becoming Leader. State: Replica: 9912d28f52f245a7bc167858c500c053, State: Running, Role: LEADER
I20260619 06:27:15.702669 481 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9912d28f52f245a7bc167858c500c053" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38729 } }
I20260619 06:27:15.705407 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9912d28f52f245a7bc167858c500c053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9912d28f52f245a7bc167858c500c053" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38729 } } }
I20260619 06:27:15.705894 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:15.705724 483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9912d28f52f245a7bc167858c500c053. Latest consensus state: current_term: 1 leader_uuid: "9912d28f52f245a7bc167858c500c053" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9912d28f52f245a7bc167858c500c053" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38729 } } }
I20260619 06:27:15.706015 483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053 [sys.catalog]: This master's current role is: LEADER
Jun 19 06:27:15 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:15.711108 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:15.716516 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:15.720448 478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:15.678922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36758 (local address 127.0.97.126:38729)
0619 06:27:15.679693 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:15.679702 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:15.680954 (+ 1252us) server_negotiation.cc:407] Connection header received
0619 06:27:15.690947 (+ 9993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:15.690965 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:15.691285 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:15.691623 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:15.694601 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:15.696112 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:15.699580 (+ 3468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:15.700940 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:15.701209 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:15.709884 (+ 8675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:15.709895 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:15.709912 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:15.709952 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:15.714921 (+ 4969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:15.715957 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:15.715966 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:15.715973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:15.716169 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:15.716767 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:15.716772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:15.716776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:15.717474 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:15.717781 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:15.718730 (+ 949us) server_negotiation.cc:299] Negotiation successful
0619 06:27:15.719169 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":359,"threads_started":1}
I20260619 06:27:15.725695 493 catalog_manager.cc:1347] Generated new cluster ID: 620399cbd0be43c2b21746480b9e2bb0
I20260619 06:27:15.725803 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:15.757006 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:15.758301 493 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:15.772975 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9912d28f52f245a7bc167858c500c053: Generated new TSK 0
I20260619 06:27:15.774071 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:15.822238 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38729
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:16.175374 502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:16.175652 502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:16.175843 502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:16.183494 502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:16.183624 502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:16.183827 502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:16.193560 502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:16.194763 502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:16.196116 502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:16.209764 507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:16.209736 508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:16.209944 510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:16.210718 502 server_base.cc:1034] running on GCE node
I20260619 06:27:16.211316 502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:16.212184 502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:16.213403 502 hybrid_clock.cc:648] HybridClock initialized: now 1781850436213342 us; error 48 us; skew 500 ppm
I20260619 06:27:16.213743 502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:16 dist-test-slave-4t1f krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:16.216930 502 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:16.218461 502 webserver.cc:460] Webserver started at http://127.0.97.65:42811/ using document root <none> and password file <none>
I20260619 06:27:16.219142 502 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:16.219281 502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:16.219595 502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:16.222661 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "07575facc4c14500b59d673fb3374071"
format_stamp: "Formatted at 2026-06-19 06:27:16 on dist-test-slave-4t1f"
server_key: "32095abee1decc16584e85d3b0819828"
server_key_iv: "e63d2548c06b1e5ca641e960477e78fe"
server_key_version: "encryptionkey@0"
I20260619 06:27:16.223450 502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "07575facc4c14500b59d673fb3374071"
format_stamp: "Formatted at 2026-06-19 06:27:16 on dist-test-slave-4t1f"
server_key: "32095abee1decc16584e85d3b0819828"
server_key_iv: "e63d2548c06b1e5ca641e960477e78fe"
server_key_version: "encryptionkey@0"
I20260619 06:27:16.228668 502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:27:16.231900 517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:16.233148 502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:16.233294 502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "07575facc4c14500b59d673fb3374071"
format_stamp: "Formatted at 2026-06-19 06:27:16 on dist-test-slave-4t1f"
server_key: "32095abee1decc16584e85d3b0819828"
server_key_iv: "e63d2548c06b1e5ca641e960477e78fe"
server_key_version: "encryptionkey@0"
I20260619 06:27:16.233494 502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:16.269794 502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:16.273720 502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:16.273999 502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:16.274901 502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:16.276578 502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:16.276654 502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:16.276810 502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:16.276870 502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:16.310333 502 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:33425
I20260619 06:27:16.310380 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:33425 every 8 connection(s)
I20260619 06:27:16.311573 502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:16.320407 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 502
I20260619 06:27:16.320622 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:16.321403 389 external_mini_cluster.cc:1428] Setting key 18237094cbf4e63c7264aff99aabb202
Jun 19 06:27:16 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:16.331466 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:16.313977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58017 (local address 127.0.97.126:38729)
0619 06:27:16.314369 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:16.314373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:16.315328 (+ 955us) server_negotiation.cc:407] Connection header received
0619 06:27:16.317008 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:16.317012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:16.317122 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:16.317214 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:16.318647 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.319313 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.320895 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.321580 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.321708 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:16.325615 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:16.325624 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:16.325632 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:16.325674 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:16.328498 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:16.329412 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:16.329415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:16.329418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:16.329520 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:16.329949 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:16.329952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:16.329954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:16.330166 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:16.330264 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:16.331069 (+ 805us) server_negotiation.cc:299] Negotiation successful
0619 06:27:16.331211 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":194,"threads_started":1}
I20260619 06:27:16.333015 633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:16.314542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38729 (local address 127.0.97.65:58017)
0619 06:27:16.315187 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:16.315220 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:16.316758 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:16.317412 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:16.317420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:16.317898 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:16.318427 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:16.318442 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.319459 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.319462 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:16.320770 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:16.320780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.321771 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.321776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:16.322481 (+ 705us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:16.323177 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:16.323198 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:16.325423 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:16.328630 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:16.328641 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:16.328659 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:16.329294 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:16.329623 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:16.329630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:16.329637 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:16.329851 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:16.330278 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:16.330288 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:16.330708 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 06:27:16.331006 (+ 298us) client_negotiation.cc:240] Negotiation successful
0619 06:27:16.331565 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":269,"threads_started":1}
I20260619 06:27:16.334506 630 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38729
I20260619 06:27:16.334805 630 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:16.335464 630 heartbeater.cc:511] Master 127.0.97.126:38729 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260619 06:27:16.337723 441 ts_manager.cc:194] Registered new tserver with Master: 07575facc4c14500b59d673fb3374071 (127.0.97.65:33425)
I20260619 06:27:16.339084 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:58017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:27:16.358342 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38729
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:16.684578 636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:16.684919 636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:16.685101 636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:16.692698 636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:16.692808 636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:16.692935 636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:16.702993 636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:16.704196 636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:16.705549 636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:16.718238 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:16.718356 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:16.718343 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:16.719350 636 server_base.cc:1034] running on GCE node
I20260619 06:27:16.719883 636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:16.721282 636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:16.722494 636 hybrid_clock.cc:648] HybridClock initialized: now 1781850436722463 us; error 51 us; skew 500 ppm
I20260619 06:27:16.722815 636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:16 dist-test-slave-4t1f krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:16.726092 636 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:16.727371 636 webserver.cc:460] Webserver started at http://127.0.97.66:38325/ using document root <none> and password file <none>
I20260619 06:27:16.727998 636 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:16.728120 636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:16.728417 636 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:16.731227 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "9efacec37d7b432a9459fe4f5801ffdf"
format_stamp: "Formatted at 2026-06-19 06:27:16 on dist-test-slave-4t1f"
server_key: "92f1d5fc51e8b3fbafddc76ba333ff5c"
server_key_iv: "87d5a066142230045bc05dfa97c89909"
server_key_version: "encryptionkey@0"
I20260619 06:27:16.731992 636 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "9efacec37d7b432a9459fe4f5801ffdf"
format_stamp: "Formatted at 2026-06-19 06:27:16 on dist-test-slave-4t1f"
server_key: "92f1d5fc51e8b3fbafddc76ba333ff5c"
server_key_iv: "87d5a066142230045bc05dfa97c89909"
server_key_version: "encryptionkey@0"
I20260619 06:27:16.736729 636 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:27:16.739879 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:16.741168 636 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260619 06:27:16.741292 636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "9efacec37d7b432a9459fe4f5801ffdf"
format_stamp: "Formatted at 2026-06-19 06:27:16 on dist-test-slave-4t1f"
server_key: "92f1d5fc51e8b3fbafddc76ba333ff5c"
server_key_iv: "87d5a066142230045bc05dfa97c89909"
server_key_version: "encryptionkey@0"
I20260619 06:27:16.741448 636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:16.767997 636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:16.772210 636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:16.772437 636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:16.773339 636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:16.774549 636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:16.774652 636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:16.774798 636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:16.774868 636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:16.808802 636 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:35175
I20260619 06:27:16.808885 763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:35175 every 8 connection(s)
I20260619 06:27:16.810055 636 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:16.813186 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 636
I20260619 06:27:16.813362 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:16.813858 389 external_mini_cluster.cc:1428] Setting key b8dbffd67bc299d185f7ed418919d576
Jun 19 06:27:16 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260619 06:27:16.831898 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:16.812517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58141 (local address 127.0.97.126:38729)
0619 06:27:16.812698 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:16.812703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:16.814557 (+ 1854us) server_negotiation.cc:407] Connection header received
0619 06:27:16.816773 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:16.816781 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:16.816874 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:16.816981 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:16.818457 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.819583 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.821177 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.821573 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.821670 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:16.825826 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:16.825831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:16.825834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:16.825864 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:16.828320 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:16.829016 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:16.829025 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:16.829028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:16.829201 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:16.830084 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:16.830089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:16.830092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:16.830345 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:16.830493 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:16.831245 (+ 752us) server_negotiation.cc:299] Negotiation successful
0619 06:27:16.831458 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260619 06:27:16.832937 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:16.813632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38729 (local address 127.0.97.66:58141)
0619 06:27:16.814393 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:16.814436 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:16.816529 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:16.817261 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:16.817269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:16.817737 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:16.818270 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:16.818286 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.819723 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.819727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:16.820980 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:16.820997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:16.821787 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:16.821796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:16.822132 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:16.822841 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:16.822874 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:16.825604 (+ 2730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:16.828455 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:16.828463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:16.828479 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:16.828890 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:16.829327 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:16.829331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:16.829335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:16.829958 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:16.830478 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:16.830485 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:16.830871 (+ 386us) client_negotiation.cc:769] Sending connection context
0619 06:27:16.831159 (+ 288us) client_negotiation.cc:240] Negotiation successful
0619 06:27:16.831664 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":357,"threads_started":1}
I20260619 06:27:16.834410 764 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38729
I20260619 06:27:16.834713 764 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:16.835356 764 heartbeater.cc:511] Master 127.0.97.126:38729 requested a full tablet report, sending...
I20260619 06:27:16.836786 441 ts_manager.cc:194] Registered new tserver with Master: 9efacec37d7b432a9459fe4f5801ffdf (127.0.97.66:35175)
I20260619 06:27:16.837759 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:58141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:16.852113 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38729
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:17.171969 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:17.172219 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:17.172312 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:17.179414 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:17.179507 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:17.179622 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:17.189812 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38729
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:17.190977 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:17.192283 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:17.205713 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:17.205760 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:17.205936 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:17.206794 769 server_base.cc:1034] running on GCE node
I20260619 06:27:17.207345 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:17.208166 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:17.209348 769 hybrid_clock.cc:648] HybridClock initialized: now 1781850437209315 us; error 26 us; skew 500 ppm
I20260619 06:27:17.209635 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:17.212545 769 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:17.213968 769 webserver.cc:460] Webserver started at http://127.0.97.67:40571/ using document root <none> and password file <none>
I20260619 06:27:17.214607 769 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:17.214744 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:17.215049 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:17.217844 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "f23e0c95e8234224bf95d61a18ad643e"
format_stamp: "Formatted at 2026-06-19 06:27:17 on dist-test-slave-4t1f"
server_key: "d7a82c62fd9025fe5d011ef3dfbcb2e7"
server_key_iv: "bff86aea46d5f14d71d27298ddbf5f0e"
server_key_version: "encryptionkey@0"
I20260619 06:27:17.218966 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "f23e0c95e8234224bf95d61a18ad643e"
format_stamp: "Formatted at 2026-06-19 06:27:17 on dist-test-slave-4t1f"
server_key: "d7a82c62fd9025fe5d011ef3dfbcb2e7"
server_key_iv: "bff86aea46d5f14d71d27298ddbf5f0e"
server_key_version: "encryptionkey@0"
I20260619 06:27:17.223555 769 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:17.226980 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:17.228147 769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:17.228327 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "f23e0c95e8234224bf95d61a18ad643e"
format_stamp: "Formatted at 2026-06-19 06:27:17 on dist-test-slave-4t1f"
server_key: "d7a82c62fd9025fe5d011ef3dfbcb2e7"
server_key_iv: "bff86aea46d5f14d71d27298ddbf5f0e"
server_key_version: "encryptionkey@0"
I20260619 06:27:17.228507 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:17.246274 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:17.250361 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:17.250635 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:17.251919 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:17.253240 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:17.253336 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:17.253484 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:17.253556 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:17.287211 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:34319
I20260619 06:27:17.287262 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:34319 every 8 connection(s)
I20260619 06:27:17.288432 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:17.296978 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 769
I20260619 06:27:17.297185 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:17.297729 389 external_mini_cluster.cc:1428] Setting key fd820648d7ba0fd4772b34d9f59698cd
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:17.307914 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:17.290855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:57253 (local address 127.0.97.126:38729)
0619 06:27:17.291017 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:17.291022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:17.292057 (+ 1035us) server_negotiation.cc:407] Connection header received
0619 06:27:17.293883 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:17.293889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:17.293989 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:17.294099 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:17.295476 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.296242 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.298628 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.299200 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.299347 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:17.303023 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:17.303034 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:17.303038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:17.303068 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:17.305483 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.306108 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.306112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.306114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.306227 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.306581 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.306584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.306586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.306823 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:17.306929 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:17.307507 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 06:27:17.307656 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:27:17.309294 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:17.291235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38729 (local address 127.0.97.67:57253)
0619 06:27:17.291905 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:17.291945 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:17.293617 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:17.294321 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:17.294329 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:17.294805 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:17.295311 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:17.295327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.296400 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.296410 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:17.298357 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:17.298374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.299461 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.299470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:17.299797 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:17.300558 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:17.300583 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:17.302810 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:17.305598 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:17.305604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:17.305615 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:17.306006 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:17.306329 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:17.306332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:17.306334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:17.306487 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:17.306924 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:17.306930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:17.307210 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 06:27:17.307430 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 06:27:17.307781 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":250,"threads_started":1}
I20260619 06:27:17.310693 897 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38729
I20260619 06:27:17.310999 897 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:17.311831 897 heartbeater.cc:511] Master 127.0.97.126:38729 requested a full tablet report, sending...
I20260619 06:27:17.313401 441 ts_manager.cc:194] Registered new tserver with Master: f23e0c95e8234224bf95d61a18ad643e (127.0.97.67:34319)
I20260619 06:27:17.314181 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:57253
I20260619 06:27:17.315179 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260619 06:27:17.341933 630 heartbeater.cc:503] Master 127.0.97.126:38729 was elected leader, sending a full tablet report...
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:17.358695 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:17.343877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.0.97.65:33425)
0619 06:27:17.344540 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:17.344550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:17.344580 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:27:17.344773 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:17.344782 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:17.344987 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:17.345186 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:17.346530 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.347507 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.349123 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.349659 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.349815 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:17.353518 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:17.353528 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:17.353541 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:17.353568 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:17.356064 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.356697 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.356704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.356706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.356814 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.357268 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.357275 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.357278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.357594 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:17.357901 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:17.358138 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 06:27:17.358380 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":377,"threads_started":1}
W20260619 06:27:17.359840 545 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32906
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:17.399768 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:17.386927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32918 (local address 127.0.97.65:33425)
0619 06:27:17.387150 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:17.387159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:17.387401 (+ 242us) server_negotiation.cc:407] Connection header received
0619 06:27:17.387596 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:17.387602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:17.387678 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:17.387779 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:17.389054 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.389750 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.390940 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.391345 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.391477 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:17.395109 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:17.395113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:17.395117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:17.395144 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:17.397411 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.398030 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.398034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.398036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.398143 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.398509 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.398515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.398518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.398770 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:17.398993 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:17.399274 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 06:27:17.399484 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:27:17.401171 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 502
I20260619 06:27:17.412945 904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:17.402029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.0.97.65:33425)
0619 06:27:17.402239 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:17.402247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:17.402507 (+ 260us) server_negotiation.cc:407] Connection header received
0619 06:27:17.402602 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:17.402606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:17.402654 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:17.402727 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:17.403829 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.404580 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.405881 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.406251 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.406344 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:17.408236 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:17.408240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:17.408243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:17.408268 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:17.410611 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.411104 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.411107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.411110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.411216 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.411485 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.411487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.411489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.411682 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:17.411833 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:17.412048 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 06:27:17.412173 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:27:17.413637 625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:17.496443 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 502
I20260619 06:27:17.512233 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 636
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:27:17.527611 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:17.513006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50936 (local address 127.0.97.66:35175)
0619 06:27:17.513653 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:17.513662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:17.513690 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:27:17.513766 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:17.513773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:17.513981 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:17.514139 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:17.515264 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.516225 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.517629 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.518163 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.518302 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:17.522245 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:17.522255 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:17.522268 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:17.522296 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:17.525135 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.525764 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.525771 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.525773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.525885 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.526353 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.526357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.526360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.526684 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:17.526890 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:17.527125 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:27:17.527322 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":355,"threads_started":1}
I20260619 06:27:17.528838 759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:17.608644 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 636
I20260619 06:27:17.630002 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 769
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[396](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:27:17.645421 913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:17.631019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.0.97.67:34319)
0619 06:27:17.631660 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:17.631666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:17.631741 (+ 75us) server_negotiation.cc:407] Connection header received
0619 06:27:17.631816 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:17.631823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:17.632033 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:17.632160 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:17.633464 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.634475 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.635606 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:17.636142 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:17.636249 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:17.640206 (+ 3957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:17.640215 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:17.640229 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:17.640262 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:17.643071 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.643615 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.643621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.643623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.643747 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:17.644022 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:17.644025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:17.644027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:17.644314 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:17.644540 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:17.644747 (+ 207us) server_negotiation.cc:299] Negotiation successful
0619 06:27:17.644918 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":316,"threads_started":1}
I20260619 06:27:17.646668 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:17.733853 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 769
I20260619 06:27:17.749516 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 409
I20260619 06:27:17.750638 471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:17.849411 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 409
2026-06-19T06:27:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3840 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[918](info): set up 2 sockets
Jun 19 06:27:17 dist-test-slave-4t1f krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260619 06:27:19.001827 389 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.001s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260619 06:27:19.217588 389 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.216s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260619 06:27:19.369258 389 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.151s user 0.000s sys 0.004s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:19 dist-test-slave-4t1f krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:19.483880 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44513
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:19.787689 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:19.788025 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:19.788204 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:19.795138 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:19.795269 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:19.795372 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:19.795476 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:19.795540 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:19.804773 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:19.806013 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:19.807260 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:19.818048 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:19.818215 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:19.818312 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:19.819923 931 server_base.cc:1034] running on GCE node
I20260619 06:27:19.820660 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:19.822059 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:19.823280 931 hybrid_clock.cc:648] HybridClock initialized: now 1781850439823249 us; error 47 us; skew 500 ppm
I20260619 06:27:19.823604 931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:19 dist-test-slave-4t1f krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:19.829808 931 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:19.831300 931 webserver.cc:460] Webserver started at http://127.0.97.126:37287/ using document root <none> and password file <none>
I20260619 06:27:19.831921 931 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:19.832017 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:19.832257 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:19.834972 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "d19989ab6e6c46a8ae69c10f304813f3"
format_stamp: "Formatted at 2026-06-19 06:27:19 on dist-test-slave-4t1f"
server_key: "7140887ee32c79e3c9557d5a04d00ba4"
server_key_iv: "60b3a7f8d26ef06566576d25c743abcd"
server_key_version: "encryptionkey@0"
I20260619 06:27:19.835639 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "d19989ab6e6c46a8ae69c10f304813f3"
format_stamp: "Formatted at 2026-06-19 06:27:19 on dist-test-slave-4t1f"
server_key: "7140887ee32c79e3c9557d5a04d00ba4"
server_key_iv: "60b3a7f8d26ef06566576d25c743abcd"
server_key_version: "encryptionkey@0"
I20260619 06:27:19.840382 931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260619 06:27:19.843560 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:19.844754 931 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:27:19.844929 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "d19989ab6e6c46a8ae69c10f304813f3"
format_stamp: "Formatted at 2026-06-19 06:27:19 on dist-test-slave-4t1f"
server_key: "7140887ee32c79e3c9557d5a04d00ba4"
server_key_iv: "60b3a7f8d26ef06566576d25c743abcd"
server_key_version: "encryptionkey@0"
I20260619 06:27:19.845155 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:19.862178 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:19.865820 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:19.866103 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:19.884243 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44513
I20260619 06:27:19.884275 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44513 every 8 connection(s)
I20260619 06:27:19.885612 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:19.887946 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 931
I20260619 06:27:19.888204 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:19.888998 389 external_mini_cluster.cc:1428] Setting key 5b6aa254c90653c9e37f57702efa218e
I20260619 06:27:19.889886 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:27:19 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:19.903733 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3: Bootstrap starting.
I20260619 06:27:19.906659 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:19.907559 998 log.cc:824] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:19.910331 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3: No bootstrap required, opened a new log
I20260619 06:27:19.910593 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:19.890990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.0.97.126:44513)
0619 06:27:19.891712 (+ 722us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:19.891722 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:19.891760 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:27:19.893259 (+ 1499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:19.893279 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:19.893648 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:19.893967 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:19.895613 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:19.896904 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:19.898630 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:19.899268 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:19.899397 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:19.903779 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:19.903787 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:19.903800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:19.903835 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:19.906773 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:19.907378 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:19.907386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:19.907400 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:19.907545 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:19.907954 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:19.907960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:19.907964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:19.908464 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:19.908755 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:19.909244 (+ 489us) server_negotiation.cc:299] Negotiation successful
0619 06:27:19.909620 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":327,"threads_started":1}
I20260619 06:27:19.917945 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19989ab6e6c46a8ae69c10f304813f3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44513 } }
I20260619 06:27:19.918275 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:19.918386 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19989ab6e6c46a8ae69c10f304813f3, State: Initialized, Role: FOLLOWER
I20260619 06:27:19.918959 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19989ab6e6c46a8ae69c10f304813f3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44513 } }
I20260619 06:27:19.919212 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:19.919353 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:19.919525 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:19.921547 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19989ab6e6c46a8ae69c10f304813f3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44513 } }
I20260619 06:27:19.921984 998 leader_election.cc:302] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19989ab6e6c46a8ae69c10f304813f3; no voters:
I20260619 06:27:19.922492 998 leader_election.cc:288] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:19.922610 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:19.923952 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [term 1 LEADER]: Becoming Leader. State: Replica: d19989ab6e6c46a8ae69c10f304813f3, State: Running, Role: LEADER
I20260619 06:27:19.924448 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19989ab6e6c46a8ae69c10f304813f3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44513 } }
I20260619 06:27:19.924540 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:19.926998 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19989ab6e6c46a8ae69c10f304813f3. Latest consensus state: current_term: 1 leader_uuid: "d19989ab6e6c46a8ae69c10f304813f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19989ab6e6c46a8ae69c10f304813f3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44513 } } }
I20260619 06:27:19.927361 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:19.927852 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d19989ab6e6c46a8ae69c10f304813f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19989ab6e6c46a8ae69c10f304813f3" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44513 } } }
I20260619 06:27:19.928128 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:19.928372 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:19.934351 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:19.942261 1011 catalog_manager.cc:1347] Generated new cluster ID: 666eed06d4b343279e4442d184350f78
I20260619 06:27:19.942373 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:19.950958 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:19.952126 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:19.959152 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d19989ab6e6c46a8ae69c10f304813f3: Generated new TSK 0
I20260619 06:27:19.959808 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:20.009027 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44513
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:20.361485 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:20.361734 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:20.361904 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:20.368929 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:20.369081 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:20.369256 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:20.378716 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:20.379921 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:20.381240 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:20.393977 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:20.393977 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:20.393901 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:20.394917 1024 server_base.cc:1034] running on GCE node
I20260619 06:27:20.395555 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:20.396433 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:20.397650 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781850440397614 us; error 50 us; skew 500 ppm
I20260619 06:27:20.397948 1024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:20 dist-test-slave-4t1f krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:20.400959 1024 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:20.402261 1024 webserver.cc:460] Webserver started at http://127.0.97.65:33685/ using document root <none> and password file <none>
I20260619 06:27:20.402832 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:20.402930 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:20.403177 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:20.405819 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "8da11edb96dc4fafa170f9f948139a7e"
format_stamp: "Formatted at 2026-06-19 06:27:20 on dist-test-slave-4t1f"
server_key: "57d2da9c317ffc4a91508402ed2eaf5b"
server_key_iv: "6716c041a97424b659b9623935d32d7b"
server_key_version: "encryptionkey@0"
I20260619 06:27:20.406517 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "8da11edb96dc4fafa170f9f948139a7e"
format_stamp: "Formatted at 2026-06-19 06:27:20 on dist-test-slave-4t1f"
server_key: "57d2da9c317ffc4a91508402ed2eaf5b"
server_key_iv: "6716c041a97424b659b9623935d32d7b"
server_key_version: "encryptionkey@0"
I20260619 06:27:20.411353 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:27:20.414529 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:20.415742 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:27:20.415872 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "8da11edb96dc4fafa170f9f948139a7e"
format_stamp: "Formatted at 2026-06-19 06:27:20 on dist-test-slave-4t1f"
server_key: "57d2da9c317ffc4a91508402ed2eaf5b"
server_key_iv: "6716c041a97424b659b9623935d32d7b"
server_key_version: "encryptionkey@0"
I20260619 06:27:20.416025 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:20.443034 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:20.447175 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:20.447427 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:20.448287 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:20.449704 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:20.449776 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:20.449863 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:20.449906 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:20.482327 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41841
I20260619 06:27:20.482337 1151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41841 every 8 connection(s)
I20260619 06:27:20.483633 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:20.491951 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 1024
I20260619 06:27:20.492166 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:20.492933 389 external_mini_cluster.cc:1428] Setting key 7df8f0b61b55d660bb7aae28c7048571
Jun 19 06:27:20 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:20.506263 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:20.486088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41825 (local address 127.0.97.126:44513)
0619 06:27:20.486503 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:20.486508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:20.487485 (+ 977us) server_negotiation.cc:407] Connection header received
0619 06:27:20.489333 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:20.489338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:20.489409 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:20.489500 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:20.490920 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.491659 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.493888 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.494299 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.494450 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:20.499321 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:20.499328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:20.499334 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:20.499376 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:20.502969 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:20.503658 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:20.503667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:20.503673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:20.503817 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:20.504211 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:20.504219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:20.504225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:20.504591 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:20.504755 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:20.505506 (+ 751us) server_negotiation.cc:299] Negotiation successful
0619 06:27:20.505763 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
I20260619 06:27:20.506767 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:20.486631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44513 (local address 127.0.97.65:41825)
0619 06:27:20.487317 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:20.487356 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:20.489051 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:20.489737 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:20.489745 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:20.490224 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:20.490751 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:20.490767 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.491809 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.491828 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:20.493717 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:20.493732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.494528 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.494537 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:20.494822 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:20.495519 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:20.495549 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:20.499045 (+ 3496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:20.503115 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:20.503122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:20.503134 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:20.503527 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:20.503926 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:20.503929 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:20.503931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:20.504094 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:20.504720 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:20.504726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:20.505029 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 06:27:20.505342 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 06:27:20.505721 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":287,"threads_started":1}
I20260619 06:27:20.508502 1152 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44513
I20260619 06:27:20.508805 1152 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:20.509541 1152 heartbeater.cc:511] Master 127.0.97.126:44513 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260619 06:27:20.511690 963 ts_manager.cc:194] Registered new tserver with Master: 8da11edb96dc4fafa170f9f948139a7e (127.0.97.65:41841)
I20260619 06:27:20.513185 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:41825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:27:20.533620 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44513
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:20.836911 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:20.837226 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:20.837395 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:20.844273 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:20.844420 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:20.844607 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:20.854038 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:20.855253 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:20.856499 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:20.868976 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:20.869133 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:20.869074 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:20.869974 1158 server_base.cc:1034] running on GCE node
I20260619 06:27:20.870523 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:20.871399 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:20.872620 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781850440872570 us; error 39 us; skew 500 ppm
I20260619 06:27:20.872907 1158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:20 dist-test-slave-4t1f krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:20.875916 1158 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:20.877295 1158 webserver.cc:460] Webserver started at http://127.0.97.66:44175/ using document root <none> and password file <none>
I20260619 06:27:20.877919 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:20.878055 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:20.878360 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:20.881331 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "211cab7442904faba5dc9bf64d70a172"
format_stamp: "Formatted at 2026-06-19 06:27:20 on dist-test-slave-4t1f"
server_key: "0322aa1dea34247efb15611be43c4a0b"
server_key_iv: "e231149d7ebc6afe2ef0dae8d7a207de"
server_key_version: "encryptionkey@0"
I20260619 06:27:20.882113 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "211cab7442904faba5dc9bf64d70a172"
format_stamp: "Formatted at 2026-06-19 06:27:20 on dist-test-slave-4t1f"
server_key: "0322aa1dea34247efb15611be43c4a0b"
server_key_iv: "e231149d7ebc6afe2ef0dae8d7a207de"
server_key_version: "encryptionkey@0"
I20260619 06:27:20.887070 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:27:20.890388 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:20.891673 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:27:20.891850 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "211cab7442904faba5dc9bf64d70a172"
format_stamp: "Formatted at 2026-06-19 06:27:20 on dist-test-slave-4t1f"
server_key: "0322aa1dea34247efb15611be43c4a0b"
server_key_iv: "e231149d7ebc6afe2ef0dae8d7a207de"
server_key_version: "encryptionkey@0"
I20260619 06:27:20.892051 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:20.912048 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:20.916116 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:20.916385 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:20.917304 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:20.918486 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:20.918592 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:20.918737 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:20.918805 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:20.950184 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41629
I20260619 06:27:20.950238 1285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41629 every 8 connection(s)
I20260619 06:27:20.951408 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:20.957795 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 1158
I20260619 06:27:20.957988 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:20.958417 389 external_mini_cluster.cc:1428] Setting key 29088037c01e0e54d13f4b31ce166021
Jun 19 06:27:20 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:20.970391 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:20.953913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51851 (local address 127.0.97.126:44513)
0619 06:27:20.954082 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:20.954086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:20.955127 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 06:27:20.956836 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:20.956840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:20.956937 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:20.957021 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:20.958634 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.959679 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.961196 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.961613 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.961745 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:20.964974 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:20.964979 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:20.964982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:20.965011 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:20.967502 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:20.968214 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:20.968221 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:20.968224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:20.968351 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:20.968756 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:20.968763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:20.968766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:20.969022 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:20.969209 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:20.969837 (+ 628us) server_negotiation.cc:299] Negotiation successful
0619 06:27:20.970045 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:27:20.971355 1288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:20.954316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44513 (local address 127.0.97.66:51851)
0619 06:27:20.954967 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:20.955006 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:20.956595 (+ 1589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:20.957280 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:20.957288 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:20.957777 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:20.958420 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:20.958441 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.959824 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.959834 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:20.960991 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:20.961002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:20.961824 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:20.961832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:20.962049 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:20.962716 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:20.962736 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:20.964753 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:20.967663 (+ 2910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:20.967672 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:20.967687 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:20.968084 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:20.968459 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:20.968466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:20.968470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:20.968649 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:20.969179 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:20.969187 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:20.969498 (+ 311us) client_negotiation.cc:769] Sending connection context
0619 06:27:20.969756 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 06:27:20.970266 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":261,"threads_started":1}
I20260619 06:27:20.972980 1286 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44513
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260619 06:27:20.973382 1286 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:20.974288 1286 heartbeater.cc:511] Master 127.0.97.126:44513 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260619 06:27:20.975849 963 ts_manager.cc:194] Registered new tserver with Master: 211cab7442904faba5dc9bf64d70a172 (127.0.97.66:41629)
I20260619 06:27:20.976613 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:51851
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:20.995669 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44513
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:21.297058 1291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:21.297334 1291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:21.297489 1291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:21.304360 1291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:21.304452 1291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:21.304670 1291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:21.313771 1291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44513
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:21.314960 1291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:21.316234 1291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:21.328889 1299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:21.328943 1296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:21.328986 1297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:21.329939 1291 server_base.cc:1034] running on GCE node
I20260619 06:27:21.330523 1291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:21.331429 1291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:21.332657 1291 hybrid_clock.cc:648] HybridClock initialized: now 1781850441332613 us; error 56 us; skew 500 ppm
I20260619 06:27:21.333048 1291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:21.336028 1291 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:21.337483 1291 webserver.cc:460] Webserver started at http://127.0.97.67:43983/ using document root <none> and password file <none>
I20260619 06:27:21.338130 1291 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:21.338271 1291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:21.338567 1291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:21.341358 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "d2fff019364241169a30bc496c4a0e62"
format_stamp: "Formatted at 2026-06-19 06:27:21 on dist-test-slave-4t1f"
server_key: "6a3466286831be5736428bac27dfac7e"
server_key_iv: "51f2de6f285839d1ddcafbf9ce1782c0"
server_key_version: "encryptionkey@0"
I20260619 06:27:21.342093 1291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "d2fff019364241169a30bc496c4a0e62"
format_stamp: "Formatted at 2026-06-19 06:27:21 on dist-test-slave-4t1f"
server_key: "6a3466286831be5736428bac27dfac7e"
server_key_iv: "51f2de6f285839d1ddcafbf9ce1782c0"
server_key_version: "encryptionkey@0"
I20260619 06:27:21.346850 1291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:21.350358 1306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:21.351672 1291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:21.351845 1291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "d2fff019364241169a30bc496c4a0e62"
format_stamp: "Formatted at 2026-06-19 06:27:21 on dist-test-slave-4t1f"
server_key: "6a3466286831be5736428bac27dfac7e"
server_key_iv: "51f2de6f285839d1ddcafbf9ce1782c0"
server_key_version: "encryptionkey@0"
I20260619 06:27:21.352056 1291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:21.379232 1291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:21.383572 1291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:21.383872 1291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:21.384807 1291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:21.386080 1291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:21.386178 1291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:21.386327 1291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:21.386397 1291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:21.419797 1291 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:45415
I20260619 06:27:21.419879 1418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:45415 every 8 connection(s)
I20260619 06:27:21.421023 1291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:21.430267 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 1291
I20260619 06:27:21.430478 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:21.430943 389 external_mini_cluster.cc:1428] Setting key 401e4c02421b947d1c68a1860df58654
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:21.441603 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.423592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42799 (local address 127.0.97.126:44513)
0619 06:27:21.423764 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.423769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.425580 (+ 1811us) server_negotiation.cc:407] Connection header received
0619 06:27:21.427440 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.427445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.427534 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.427638 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.429264 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.430097 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.431906 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.432456 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.432554 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.436208 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:21.436213 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:21.436217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:21.436248 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:21.438937 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.439596 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.439600 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.439603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.439725 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.440125 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.440131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.440133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.440379 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:21.440483 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.441198 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.441358 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:27:21.442452 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.424091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44513 (local address 127.0.97.67:42799)
0619 06:27:21.425419 (+ 1328us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:21.425461 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:21.427177 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:21.427885 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:21.427893 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:21.428400 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:21.429043 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.429104 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.430279 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.430284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.431722 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.431741 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.432668 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.432673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.433002 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.433847 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:21.433877 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:21.436018 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:21.439051 (+ 3033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:21.439057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:21.439068 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:21.439448 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:21.439854 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:21.439857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:21.439859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:21.440025 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:21.440475 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:21.440480 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:21.440760 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 06:27:21.440980 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 06:27:21.441402 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1029,"thread_start_us":722,"threads_started":1}
I20260619 06:27:21.443837 1419 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44513
I20260619 06:27:21.444082 1419 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:21.444823 1419 heartbeater.cc:511] Master 127.0.97.126:44513 requested a full tablet report, sending...
I20260619 06:27:21.446462 963 ts_manager.cc:194] Registered new tserver with Master: d2fff019364241169a30bc496c4a0e62 (127.0.97.67:45415)
I20260619 06:27:21.447618 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:42799
I20260619 06:27:21.458223 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:27:21.476241 1154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.464086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36730 (local address 127.0.97.126:44513)
0619 06:27:21.464268 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.464274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.464738 (+ 464us) server_negotiation.cc:407] Connection header received
0619 06:27:21.464864 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.464868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.464922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.464998 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.466386 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.467079 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.468292 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.468694 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.468784 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.471150 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:21.471154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:21.471158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:21.471184 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:21.474149 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.474707 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.474711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.474714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.474820 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.475111 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.475114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.475116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.475406 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:21.475587 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.475868 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.476026 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:27:21.482074 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:21.488539 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:21.515003 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.500667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41841 (local address 127.0.0.1:34042)
0619 06:27:21.501557 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:21.501575 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:21.501704 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:21.504203 (+ 2499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:21.504210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:21.504244 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:21.504753 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.504768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.507239 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.507247 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.510824 (+ 3577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.510844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.513637 (+ 2793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.513647 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.513914 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.513981 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 06:27:21.514132 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 06:27:21.514378 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":136,"spinlock_wait_cycles":768,"thread_start_us":1430,"threads_started":1}
I20260619 06:27:21.516350 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.501929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.0.97.65:41841)
0619 06:27:21.503300 (+ 1371us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.503308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.503336 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:27:21.503445 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.503451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.503708 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.503922 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:21.504989 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.507040 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.511073 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.513032 (+ 1959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.513200 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.513496 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.515637 (+ 2141us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.515913 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":1416,"threads_started":1}
I20260619 06:27:21.516729 1152 heartbeater.cc:503] Master 127.0.97.126:44513 was elected leader, sending a full tablet report...
I20260619 06:27:21.517668 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.500301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:45415 (local address 127.0.0.1:53182)
0619 06:27:21.501406 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:21.501483 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:21.501704 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:21.503237 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:21.503244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:21.503272 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:21.503604 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.503616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.507295 (+ 3679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.507302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.512394 (+ 5092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.512414 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.516727 (+ 4313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.516733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.516977 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.517015 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:27:21.517133 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:27:21.517252 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":136,"spinlock_wait_cycles":2944,"thread_start_us":3346,"threads_started":1}
I20260619 06:27:21.517958 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.501217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.0.97.67:45415)
0619 06:27:21.502637 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.502647 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.502682 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:27:21.502773 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.502784 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.503029 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.503225 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:21.503804 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.507111 (+ 3307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.512633 (+ 5522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.516203 (+ 3570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.516364 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.517330 (+ 966us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.517409 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.517557 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":2292,"threads_started":1}
I20260619 06:27:21.521680 1354 tablet_service.cc:1467] Processing CreateTablet for tablet 1f775b5d0cdc4d56abf98b095f89ae66 (DEFAULT_TABLE table=test-table [id=054e6e216d6d4a1395d527054f85de47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:21.521744 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 1f775b5d0cdc4d56abf98b095f89ae66 (DEFAULT_TABLE table=test-table [id=054e6e216d6d4a1395d527054f85de47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:21.523072 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f775b5d0cdc4d56abf98b095f89ae66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:21.523079 1354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f775b5d0cdc4d56abf98b095f89ae66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:21.524605 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.502763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:41629 (local address 127.0.0.1:36192)
0619 06:27:21.515282 (+ 12519us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:21.515306 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:21.515434 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:21.517775 (+ 2341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:21.517780 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:21.517818 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:21.517996 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.518007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.520046 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.520052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.521903 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.521917 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.523821 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.523827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.524009 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.524042 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:27:21.524143 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:27:21.524274 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12395,"thread_start_us":15351,"threads_started":1}
I20260619 06:27:21.526060 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.503218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.0.97.66:41629)
0619 06:27:21.516906 (+ 13688us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.516916 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.516947 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:27:21.517043 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.517050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.517307 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.517507 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:21.518151 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.519896 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.522067 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.523434 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.523557 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.523731 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.524207 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.525300 (+ 1093us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13518,"thread_start_us":13399,"threads_started":1}
I20260619 06:27:21.532641 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 1f775b5d0cdc4d56abf98b095f89ae66 (DEFAULT_TABLE table=test-table [id=054e6e216d6d4a1395d527054f85de47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:21.534086 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f775b5d0cdc4d56abf98b095f89ae66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:21.534900 1437 tablet_bootstrap.cc:492] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e: Bootstrap starting.
I20260619 06:27:21.537150 1438 tablet_bootstrap.cc:492] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62: Bootstrap starting.
I20260619 06:27:21.538861 1437 tablet_bootstrap.cc:654] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:21.540091 1437 log.cc:824] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:21.540355 1438 tablet_bootstrap.cc:654] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:21.541728 1438 log.cc:824] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:21.544344 1437 tablet_bootstrap.cc:492] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e: No bootstrap required, opened a new log
I20260619 06:27:21.544653 1437 ts_tablet_manager.cc:1397] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20260619 06:27:21.546104 1441 tablet_bootstrap.cc:492] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172: Bootstrap starting.
I20260619 06:27:21.546926 1438 tablet_bootstrap.cc:492] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62: No bootstrap required, opened a new log
I20260619 06:27:21.547160 1438 ts_tablet_manager.cc:1397] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260619 06:27:21.548888 1441 tablet_bootstrap.cc:654] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:21.550158 1441 log.cc:824] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:21.552807 1441 tablet_bootstrap.cc:492] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172: No bootstrap required, opened a new log
I20260619 06:27:21.553122 1441 ts_tablet_manager.cc:1397] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:27:21.556911 1437 raft_consensus.cc:348] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.557324 1437 raft_consensus.cc:374] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:21.557440 1437 raft_consensus.cc:729] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8da11edb96dc4fafa170f9f948139a7e, State: Initialized, Role: FOLLOWER
I20260619 06:27:21.557829 1438 raft_consensus.cc:348] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.558220 1438 raft_consensus.cc:374] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:21.558069 1437 consensus_queue.cc:260] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.558310 1438 raft_consensus.cc:729] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2fff019364241169a30bc496c4a0e62, State: Initialized, Role: FOLLOWER
I20260619 06:27:21.558998 1438 consensus_queue.cc:260] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.559908 1437 ts_tablet_manager.cc:1428] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e: Time spent starting tablet: real 0.015s user 0.003s sys 0.011s
I20260619 06:27:21.560794 1419 heartbeater.cc:503] Master 127.0.97.126:44513 was elected leader, sending a full tablet report...
I20260619 06:27:21.561426 1438 ts_tablet_manager.cc:1428] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260619 06:27:21.561646 1441 raft_consensus.cc:348] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.562036 1441 raft_consensus.cc:374] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:21.562136 1441 raft_consensus.cc:729] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211cab7442904faba5dc9bf64d70a172, State: Initialized, Role: FOLLOWER
I20260619 06:27:21.563573 1441 consensus_queue.cc:260] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.564626 1286 heartbeater.cc:503] Master 127.0.97.126:44513 was elected leader, sending a full tablet report...
I20260619 06:27:21.565237 1441 ts_tablet_manager.cc:1428] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260619 06:27:21.602015 1445 raft_consensus.cc:482] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:21.602295 1445 raft_consensus.cc:504] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.604316 1445 leader_election.cc:288] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2fff019364241169a30bc496c4a0e62 (127.0.97.67:45415), 8da11edb96dc4fafa170f9f948139a7e (127.0.97.65:41841)
I20260619 06:27:21.613272 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.604803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50601 (local address 127.0.97.67:45415)
0619 06:27:21.605111 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.605119 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.605596 (+ 477us) server_negotiation.cc:407] Connection header received
0619 06:27:21.605789 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.605795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.605871 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.605974 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:21.606753 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.607862 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.610868 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.612060 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.612191 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.612286 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.612884 (+ 598us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.613006 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 06:27:21.613274 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.604799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58503 (local address 127.0.97.65:41841)
0619 06:27:21.605004 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.605012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.605619 (+ 607us) server_negotiation.cc:407] Connection header received
0619 06:27:21.605790 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.605795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.605895 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.605980 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:21.606753 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.607870 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.610893 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.612081 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.612194 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.612286 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.612902 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.613015 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:27:21.614012 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.604590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:45415 (local address 127.0.97.66:50601)
0619 06:27:21.605462 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:21.605480 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:21.605631 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:21.606014 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:21.606020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:21.606048 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:21.606621 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.606635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.607996 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.608002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.610713 (+ 2711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.610726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.612328 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.612334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.612618 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.612735 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 06:27:21.612841 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:27:21.613003 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":338,"thread_start_us":574,"threads_started":1}
I20260619 06:27:21.614048 1446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.604653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41841 (local address 127.0.97.66:58503)
0619 06:27:21.605515 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:21.605530 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:21.605667 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:21.606014 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:21.606020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:21.606057 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:21.606582 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.606599 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.607998 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.608002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.610712 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:21.610726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.612315 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.612321 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:21.612616 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.612735 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 06:27:21.612841 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:27:21.613003 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":325,"spinlock_wait_cycles":47616,"thread_start_us":472,"threads_started":1}
I20260619 06:27:21.615083 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f775b5d0cdc4d56abf98b095f89ae66" candidate_uuid: "211cab7442904faba5dc9bf64d70a172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2fff019364241169a30bc496c4a0e62" is_pre_election: true
I20260619 06:27:21.615121 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f775b5d0cdc4d56abf98b095f89ae66" candidate_uuid: "211cab7442904faba5dc9bf64d70a172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da11edb96dc4fafa170f9f948139a7e" is_pre_election: true
I20260619 06:27:21.615552 1374 raft_consensus.cc:2454] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 211cab7442904faba5dc9bf64d70a172 in term 0.
I20260619 06:27:21.615567 1107 raft_consensus.cc:2454] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 211cab7442904faba5dc9bf64d70a172 in term 0.
I20260619 06:27:21.616259 1175 leader_election.cc:302] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 211cab7442904faba5dc9bf64d70a172, 8da11edb96dc4fafa170f9f948139a7e, d2fff019364241169a30bc496c4a0e62; no voters:
I20260619 06:27:21.616658 1445 raft_consensus.cc:2781] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:21.616787 1445 raft_consensus.cc:482] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:21.616894 1445 raft_consensus.cc:3037] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:21.619059 1445 raft_consensus.cc:504] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.619726 1445 leader_election.cc:288] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [CANDIDATE]: Term 1 election: Requested vote from peers d2fff019364241169a30bc496c4a0e62 (127.0.97.67:45415), 8da11edb96dc4fafa170f9f948139a7e (127.0.97.65:41841)
I20260619 06:27:21.620178 1374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f775b5d0cdc4d56abf98b095f89ae66" candidate_uuid: "211cab7442904faba5dc9bf64d70a172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2fff019364241169a30bc496c4a0e62"
I20260619 06:27:21.620220 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f775b5d0cdc4d56abf98b095f89ae66" candidate_uuid: "211cab7442904faba5dc9bf64d70a172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8da11edb96dc4fafa170f9f948139a7e"
I20260619 06:27:21.620361 1374 raft_consensus.cc:3037] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:21.620385 1107 raft_consensus.cc:3037] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:21.622505 1374 raft_consensus.cc:2454] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 211cab7442904faba5dc9bf64d70a172 in term 1.
I20260619 06:27:21.622509 1107 raft_consensus.cc:2454] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 211cab7442904faba5dc9bf64d70a172 in term 1.
I20260619 06:27:21.622913 1176 leader_election.cc:302] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 211cab7442904faba5dc9bf64d70a172, 8da11edb96dc4fafa170f9f948139a7e; no voters:
I20260619 06:27:21.623216 1445 raft_consensus.cc:2781] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:21.623721 1445 raft_consensus.cc:686] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [term 1 LEADER]: Becoming Leader. State: Replica: 211cab7442904faba5dc9bf64d70a172, State: Running, Role: LEADER
I20260619 06:27:21.624261 1445 consensus_queue.cc:237] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } }
I20260619 06:27:21.629477 963 catalog_manager.cc:5495] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172 reported cstate change: term changed from 0 to 1, leader changed from <none> to 211cab7442904faba5dc9bf64d70a172 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "211cab7442904faba5dc9bf64d70a172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2fff019364241169a30bc496c4a0e62" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 45415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8da11edb96dc4fafa170f9f948139a7e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "211cab7442904faba5dc9bf64d70a172" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 41629 } health_report { overall_health: HEALTHY } } }
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:21.670117 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.656117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.0.97.65:41841)
0619 06:27:21.656310 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.656314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.656796 (+ 482us) server_negotiation.cc:407] Connection header received
0619 06:27:21.656919 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.656922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.657018 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.657124 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.658473 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.659160 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.660462 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.660885 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.660976 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.664721 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:21.664735 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:21.664755 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:21.664799 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:21.667989 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.668558 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.668565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.668567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.668673 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.668957 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.668960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.668962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.669329 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:21.669532 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.669728 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.669866 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:27:21.672570 1420 tablet.cc:2369] T 1f775b5d0cdc4d56abf98b095f89ae66 P d2fff019364241169a30bc496c4a0e62: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260619 06:27:21.703248 1287 tablet.cc:2369] T 1f775b5d0cdc4d56abf98b095f89ae66 P 211cab7442904faba5dc9bf64d70a172: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:21.712328 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.699763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34064 (local address 127.0.97.65:41841)
0619 06:27:21.700100 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.700108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.700340 (+ 232us) server_negotiation.cc:407] Connection header received
0619 06:27:21.700473 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.700476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.700541 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.700622 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.701885 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.702579 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.703874 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.704345 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.704440 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.707806 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:21.707810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:21.707813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:21.707840 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:21.710432 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.710947 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.710951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.710953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.711092 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.711367 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.711370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.711372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.711596 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:21.711752 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.711963 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.712096 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260619 06:27:21.735848 1153 tablet.cc:2369] T 1f775b5d0cdc4d56abf98b095f89ae66 P 8da11edb96dc4fafa170f9f948139a7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:21.753595 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.741002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.0.97.65:41841)
0619 06:27:21.741254 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.741260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.741365 (+ 105us) server_negotiation.cc:407] Connection header received
0619 06:27:21.741531 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.741535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.741595 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.741677 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.742946 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.743602 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.744736 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.745188 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.745277 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.749034 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:21.749039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:21.749042 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:21.749115 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:21.751365 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.751889 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.751893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.751895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.752006 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.752332 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.752334 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.752336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.752609 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:21.752784 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.752990 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.753228 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260619 06:27:21.755259 1067 tablet_service.cc:2908] Rejecting scan request for tablet 1f775b5d0cdc4d56abf98b095f89ae66: Uninitialized: clean time has not yet been initialized
I20260619 06:27:21.758121 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1024
I20260619 06:27:21.770843 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.759072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.0.97.65:41841)
0619 06:27:21.759251 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.759259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.759568 (+ 309us) server_negotiation.cc:407] Connection header received
0619 06:27:21.759750 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.759759 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.759822 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.759919 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.760998 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.761707 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.762862 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.763286 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.763422 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.765552 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:21.765557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:21.765560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:21.765594 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:21.768483 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.769152 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.769159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.769166 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.769267 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.769615 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.769622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.769625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.769861 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:21.770159 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.770351 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.770545 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:27:21.771438 1147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:21.858508 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 1024
I20260619 06:27:21.875089 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1158
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:27:21.889505 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.875821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36198 (local address 127.0.97.66:41629)
0619 06:27:21.875997 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.876002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.876020 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:27:21.876180 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.876184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.876241 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.876324 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.877566 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.878267 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.879458 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.879940 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:21.880081 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:21.883979 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:21.883992 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:21.884007 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:21.884054 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:21.886846 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.887523 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.887530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.887532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.887672 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:21.888036 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:21.888043 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:21.888046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:21.888365 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:21.888686 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:21.888908 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:27:21.889154 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:27:21.890151 1281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:21.979985 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 1158
I20260619 06:27:21.996604 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1291
Jun 19 06:27:21 dist-test-slave-4t1f krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:27:22.011019 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:21.997293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53196 (local address 127.0.97.67:45415)
0619 06:27:21.997468 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:21.997472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:21.997490 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:27:21.997651 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:21.997655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:21.997716 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:21.997808 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:21.998994 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:21.999754 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:22.001204 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:22.001611 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:22.001722 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:22.005846 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:22.005860 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:22.005876 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:22.005912 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:22.008692 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:22.009360 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:22.009368 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:22.009370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:22.009495 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:22.009848 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:22.009851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:22.009853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:22.010185 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:22.010388 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:22.010644 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 06:27:22.010792 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:27:22.011646 1414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:22.103935 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 1291
I20260619 06:27:22.119508 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 931
I20260619 06:27:22.120648 993 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:22.223127 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 931
2026-06-19T06:27:22Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4374 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:22 dist-test-slave-4t1f krb5kdc[1467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:22 dist-test-slave-4t1f krb5kdc[1467](info): set up 2 sockets
Jun 19 06:27:22 dist-test-slave-4t1f krb5kdc[1467](info): commencing operation
krb5kdc: starting...
W20260619 06:27:23.296957 389 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:23 dist-test-slave-4t1f krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:23.462471 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45935
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:23.765058 1480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:23.765352 1480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:23.765516 1480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:23.772440 1480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:23.772559 1480 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:27:23.772634 1480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:23.772750 1480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:23.772835 1480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:23.772924 1480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:23.782243 1480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45935
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:23.783389 1480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:23.784595 1480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:23.796087 1485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:23.796167 1486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:23.796065 1488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:23.797369 1480 server_base.cc:1034] running on GCE node
I20260619 06:27:23.798149 1480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:23.799348 1480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:23.800590 1480 hybrid_clock.cc:648] HybridClock initialized: now 1781850443800534 us; error 36 us; skew 500 ppm
I20260619 06:27:23.800933 1480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:23 dist-test-slave-4t1f krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:23.804003 1480 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:23.805294 1480 webserver.cc:460] Webserver started at http://127.0.97.126:37323/ using document root <none> and password file <none>
I20260619 06:27:23.805948 1480 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:23.806031 1480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:23.806282 1480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:23.808909 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "14c5744cd70b4e9b836a0131c6a9a11d"
format_stamp: "Formatted at 2026-06-19 06:27:23 on dist-test-slave-4t1f"
server_key: "b02590bda95d48429050b4309087f74b"
server_key_iv: "c2cdf5f51e95db76604525c360bd9a53"
server_key_version: "encryptionkey@0"
I20260619 06:27:23.809706 1480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "14c5744cd70b4e9b836a0131c6a9a11d"
format_stamp: "Formatted at 2026-06-19 06:27:23 on dist-test-slave-4t1f"
server_key: "b02590bda95d48429050b4309087f74b"
server_key_iv: "c2cdf5f51e95db76604525c360bd9a53"
server_key_version: "encryptionkey@0"
I20260619 06:27:23.814393 1480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:27:23.817535 1495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:23.818801 1480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:27:23.818931 1480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "14c5744cd70b4e9b836a0131c6a9a11d"
format_stamp: "Formatted at 2026-06-19 06:27:23 on dist-test-slave-4t1f"
server_key: "b02590bda95d48429050b4309087f74b"
server_key_iv: "c2cdf5f51e95db76604525c360bd9a53"
server_key_version: "encryptionkey@0"
I20260619 06:27:23.819092 1480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:23.842044 1480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:23.846149 1480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:23.846347 1480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:23.865799 1480 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45935
I20260619 06:27:23.865824 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45935 every 8 connection(s)
I20260619 06:27:23.867050 1480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:23.870599 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:23.876798 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 1480
I20260619 06:27:23.877034 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:23.877676 389 external_mini_cluster.cc:1428] Setting key 9a0fba9783776268ba7a9e1abaaddd61
I20260619 06:27:23.880594 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d: Bootstrap starting.
I20260619 06:27:23.883823 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:23.885011 1557 log.cc:824] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:23.888026 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d: No bootstrap required, opened a new log
Jun 19 06:27:23 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:23.898319 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45935 } }
I20260619 06:27:23.898741 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:23.898849 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14c5744cd70b4e9b836a0131c6a9a11d, State: Initialized, Role: FOLLOWER
I20260619 06:27:23.899519 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45935 } }
I20260619 06:27:23.899801 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:23.899932 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:23.900094 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:23.901605 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:23.879473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.0.97.126:45935)
0619 06:27:23.880224 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:23.880234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:23.880271 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:27:23.882272 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:23.882292 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:23.882635 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:23.882914 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:23.884709 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:23.886125 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:23.887847 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:23.888552 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:23.888674 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:23.893328 (+ 4654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:23.893338 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:23.893357 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:23.893400 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:23.897280 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:23.897928 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:23.897935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:23.897942 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:23.898167 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:23.898484 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:23.898489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:23.898493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:23.899210 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:23.899468 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:23.899796 (+ 328us) server_negotiation.cc:299] Negotiation successful
0619 06:27:23.900246 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":361,"threads_started":1}
I20260619 06:27:23.902974 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45935 } }
I20260619 06:27:23.903507 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14c5744cd70b4e9b836a0131c6a9a11d; no voters:
I20260619 06:27:23.904125 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:23.904232 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:23.905258 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [term 1 LEADER]: Becoming Leader. State: Replica: 14c5744cd70b4e9b836a0131c6a9a11d, State: Running, Role: LEADER
I20260619 06:27:23.905793 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45935 } }
I20260619 06:27:23.906062 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:23.908335 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14c5744cd70b4e9b836a0131c6a9a11d. Latest consensus state: current_term: 1 leader_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45935 } } }
I20260619 06:27:23.908639 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:23.909286 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14c5744cd70b4e9b836a0131c6a9a11d" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45935 } } }
I20260619 06:27:23.909556 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:23.909678 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:23.915155 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:23.923233 1567 catalog_manager.cc:1347] Generated new cluster ID: 886c5ebea658434fb18266c8bafd8f21
I20260619 06:27:23.923331 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:23.947664 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:23.948889 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:23.961824 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d: Generated new TSK 0
I20260619 06:27:23.962803 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:24.014312 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45935
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:24.326090 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:24.326352 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:24.326448 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:24.333268 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:24.333360 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:24.333439 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:27:24.333518 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:24.342612 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:45935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:24.343696 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:24.344893 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:24.357538 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:24.357612 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:24.357584 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:24.358403 1583 server_base.cc:1034] running on GCE node
I20260619 06:27:24.358952 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:24.359783 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:24.360982 1583 hybrid_clock.cc:648] HybridClock initialized: now 1781850444360914 us; error 54 us; skew 500 ppm
I20260619 06:27:24.361292 1583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:24 dist-test-slave-4t1f krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:24.364331 1583 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:24.365759 1583 webserver.cc:460] Webserver started at http://127.0.97.65:44939/ using document root <none> and password file <none>
I20260619 06:27:24.366420 1583 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:24.366546 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:24.366850 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:24.369592 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "2c3fbdbf30254da3904a53453f47d206"
format_stamp: "Formatted at 2026-06-19 06:27:24 on dist-test-slave-4t1f"
server_key: "817faae2f8c5a0ff2c5a98b6467d8465"
server_key_iv: "b1bd4630d1dffec0ef42165851f30101"
server_key_version: "encryptionkey@0"
I20260619 06:27:24.370347 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "2c3fbdbf30254da3904a53453f47d206"
format_stamp: "Formatted at 2026-06-19 06:27:24 on dist-test-slave-4t1f"
server_key: "817faae2f8c5a0ff2c5a98b6467d8465"
server_key_iv: "b1bd4630d1dffec0ef42165851f30101"
server_key_version: "encryptionkey@0"
I20260619 06:27:24.375001 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:24.378309 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:24.379499 1583 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:27:24.379665 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "2c3fbdbf30254da3904a53453f47d206"
format_stamp: "Formatted at 2026-06-19 06:27:24 on dist-test-slave-4t1f"
server_key: "817faae2f8c5a0ff2c5a98b6467d8465"
server_key_iv: "b1bd4630d1dffec0ef42165851f30101"
server_key_version: "encryptionkey@0"
I20260619 06:27:24.379873 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:24.407948 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:24.411932 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:24.412179 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:24.414801 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:24.414916 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:24.415081 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:24.415164 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:27:24 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:24.440900 1608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:24.415791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45935 (local address 127.0.97.65:49601)
0619 06:27:24.416631 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:24.416691 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:24.419365 (+ 2674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:24.420187 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:24.420197 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:24.421127 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:24.422038 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:24.422057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.423549 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.423557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:24.426333 (+ 2776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:24.426354 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.427389 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.427400 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:24.427745 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:24.428507 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:24.428552 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:24.432445 (+ 3893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:24.436029 (+ 3584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:24.436034 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:24.436044 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:24.436740 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:24.437149 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:24.437156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:24.437165 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:24.437382 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:24.438044 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:24.438055 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:24.438373 (+ 318us) client_negotiation.cc:769] Sending connection context
0619 06:27:24.438669 (+ 296us) client_negotiation.cc:240] Negotiation successful
0619 06:27:24.439221 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":319,"threads_started":1}
I20260619 06:27:24.442418 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:24.415145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49601 (local address 127.0.97.126:45935)
0619 06:27:24.415646 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:24.415652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:24.416816 (+ 1164us) server_negotiation.cc:407] Connection header received
0619 06:27:24.419681 (+ 2865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:24.419689 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:24.419772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:24.419899 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:24.422201 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.423393 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.426568 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.427102 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.427248 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:24.432698 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:24.432702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:24.432706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:24.432734 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:24.435848 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:24.436873 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:24.436878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:24.436882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:24.437010 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:24.437498 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:24.437502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:24.437506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:24.437897 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:24.438048 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:24.441693 (+ 3645us) server_negotiation.cc:299] Negotiation successful
0619 06:27:24.441934 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":226,"threads_started":1}
I20260619 06:27:24.454540 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:45709
I20260619 06:27:24.454581 1718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:45709 every 8 connection(s)
I20260619 06:27:24.455910 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:24.458411 1719 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45935
I20260619 06:27:24.458639 1719 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:24.459406 1719 heartbeater.cc:511] Master 127.0.97.126:45935 requested a full tablet report, sending...
I20260619 06:27:24.459784 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 1583
I20260619 06:27:24.460012 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:24.460640 389 external_mini_cluster.cc:1428] Setting key ab5580c8d2ef8ad50670b29c6c57ae4f
I20260619 06:27:24.461728 1512 ts_manager.cc:194] Registered new tserver with Master: 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709)
I20260619 06:27:24.463125 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:24.449481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:48149 (local address 127.0.97.126:45935)
0619 06:27:24.449665 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:24.449672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:24.450339 (+ 667us) server_negotiation.cc:407] Connection header received
0619 06:27:24.450517 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:24.450520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:24.450572 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:24.450672 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:24.451775 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.452945 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.454783 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.455186 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.455278 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:24.457480 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:24.457486 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:24.457490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:24.457522 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:24.460502 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:24.461274 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:24.461279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:24.461283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:24.461444 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:24.461919 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:24.461924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:24.461928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:24.462218 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:24.462339 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:24.462747 (+ 408us) server_negotiation.cc:299] Negotiation successful
0619 06:27:24.462895 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:27:24.463374 1707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:24.449495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45935 (local address 127.0.97.65:48149)
0619 06:27:24.450222 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:24.450245 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:24.450402 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:24.450695 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:24.450699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:24.451268 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:24.451588 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:24.451622 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.453114 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.453119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:24.454598 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:24.454610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:24.455380 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:24.455387 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:24.455584 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:24.456218 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:24.456235 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:24.457315 (+ 1080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:24.460633 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:24.460639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:24.460642 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:24.461163 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:24.461612 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:24.461619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:24.461626 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:24.461785 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:24.462327 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:24.462332 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:24.462501 (+ 169us) client_negotiation.cc:769] Sending connection context
0619 06:27:24.462657 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 06:27:24.462949 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":386,"threads_started":1}
I20260619 06:27:24.463780 1512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:49601
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
W20260619 06:27:24.861488 389 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.97.66: real 0.395s user 0.002s sys 0.006s
I20260619 06:27:24.862197 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45935
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:25.169049 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:25.169327 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:25.169481 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:25.176344 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:25.176433 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:25.176513 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:27:25.176687 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:25.185949 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:45935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:25.187057 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:25.188287 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:25.200853 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:25.200896 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:25.200925 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:25.201705 1723 server_base.cc:1034] running on GCE node
I20260619 06:27:25.202262 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:25.203070 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:25.204296 1723 hybrid_clock.cc:648] HybridClock initialized: now 1781850445204253 us; error 39 us; skew 500 ppm
I20260619 06:27:25.204566 1723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:25 dist-test-slave-4t1f krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:25.212386 1723 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:25.213797 1723 webserver.cc:460] Webserver started at http://127.0.97.66:40169/ using document root <none> and password file <none>
I20260619 06:27:25.214450 1723 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:25.214576 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:25.214892 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:25.217583 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "d9166564c0cd46469dfd2a2cf18d63b7"
format_stamp: "Formatted at 2026-06-19 06:27:25 on dist-test-slave-4t1f"
server_key: "873aa33dc3103d42663b6e9b385e2c56"
server_key_iv: "1155aa70cbc7c9f172b0e81d4efe1d13"
server_key_version: "encryptionkey@0"
I20260619 06:27:25.218326 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "d9166564c0cd46469dfd2a2cf18d63b7"
format_stamp: "Formatted at 2026-06-19 06:27:25 on dist-test-slave-4t1f"
server_key: "873aa33dc3103d42663b6e9b385e2c56"
server_key_iv: "1155aa70cbc7c9f172b0e81d4efe1d13"
server_key_version: "encryptionkey@0"
I20260619 06:27:25.223209 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:27:25.226282 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:25.227435 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:27:25.227602 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "d9166564c0cd46469dfd2a2cf18d63b7"
format_stamp: "Formatted at 2026-06-19 06:27:25 on dist-test-slave-4t1f"
server_key: "873aa33dc3103d42663b6e9b385e2c56"
server_key_iv: "1155aa70cbc7c9f172b0e81d4efe1d13"
server_key_version: "encryptionkey@0"
I20260619 06:27:25.227816 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:25.244733 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:25.248755 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:25.248996 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:25.251401 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:25.251498 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:25.251659 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:25.251740 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:27:25 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:25.273314 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.252036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44333 (local address 127.0.97.126:45935)
0619 06:27:25.252571 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.252577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.253650 (+ 1073us) server_negotiation.cc:407] Connection header received
0619 06:27:25.255576 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.255584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.255684 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.255815 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:25.257995 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.259283 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.262540 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.262970 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.263064 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.267331 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:25.267335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:25.267338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:25.267367 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:25.270198 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.271171 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.271177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.271181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.271296 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.271705 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.271710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.271713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.271951 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:25.272092 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.272705 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.272905 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":312,"threads_started":1}
I20260619 06:27:25.274179 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.252590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45935 (local address 127.0.97.66:44333)
0619 06:27:25.253481 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:25.253531 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:25.255307 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:25.256049 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:25.256063 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:25.256831 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:25.257723 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.257744 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.259584 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.259591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.262371 (+ 2780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.262391 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.263213 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.263223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.263548 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.264253 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:25.264289 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:25.267120 (+ 2831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:25.270384 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.270392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.270404 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.271040 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.271399 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.271405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.271408 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.271588 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.272067 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:25.272074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:25.272387 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 06:27:25.272613 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 06:27:25.273056 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":422,"threads_started":1}
I20260619 06:27:25.295253 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.280078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:54905 (local address 127.0.97.126:45935)
0619 06:27:25.280225 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.280229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.281360 (+ 1131us) server_negotiation.cc:407] Connection header received
0619 06:27:25.281561 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.281564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.281618 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.281696 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:25.283017 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.283745 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.286136 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.286658 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.286799 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.289349 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:25.289356 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:25.289363 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:25.289405 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:25.292447 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.293421 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.293426 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.293430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.293550 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.293909 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.293913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.293917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.294209 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:25.294354 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.294723 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.294922 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 06:27:25.295341 1797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.280069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45935 (local address 127.0.97.66:54905)
0619 06:27:25.281235 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:25.281260 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:25.281447 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:25.281762 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:25.281770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:25.282416 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:25.282869 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.282884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.283917 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.283925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.285981 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.286000 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.287014 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.287018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.287323 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.288070 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:25.288101 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:25.289123 (+ 1022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:25.292626 (+ 3503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.292632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.292636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.293298 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.293653 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.293657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.293661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.293793 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.294320 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:25.294325 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:25.294497 (+ 172us) client_negotiation.cc:769] Sending connection context
0619 06:27:25.294622 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 06:27:25.294915 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"thread_start_us":701,"threads_started":1}
I20260619 06:27:25.307461 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:33853
I20260619 06:27:25.307536 1858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:33853 every 8 connection(s)
I20260619 06:27:25.308626 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:25.310570 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 1723
I20260619 06:27:25.310580 1859 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45935
I20260619 06:27:25.310729 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:25.310792 1859 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:25.311180 389 external_mini_cluster.cc:1428] Setting key ad108917e93a17684c1144b11274067c
I20260619 06:27:25.311419 1859 heartbeater.cc:511] Master 127.0.97.126:45935 requested a full tablet report, sending...
I20260619 06:27:25.312762 1511 ts_manager.cc:194] Registered new tserver with Master: d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66:33853)
I20260619 06:27:25.313715 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:44333
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:25.348313 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45935
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:27:25.467147 1719 heartbeater.cc:503] Master 127.0.97.126:45935 was elected leader, sending a full tablet report...
W20260619 06:27:25.656543 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:25.656869 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:25.657048 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:25.663928 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:25.664064 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:25.664218 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:27:25.664361 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:25.673446 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:45935
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:25.674641 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:25.675937 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:25.688495 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:25.688463 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:25.689191 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:25.689391 1863 server_base.cc:1034] running on GCE node
I20260619 06:27:25.690003 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:25.690748 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:25.692029 1863 hybrid_clock.cc:648] HybridClock initialized: now 1781850445691978 us; error 36 us; skew 500 ppm
I20260619 06:27:25.692261 1863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:25 dist-test-slave-4t1f krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:25.695277 1863 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:25.696446 1863 webserver.cc:460] Webserver started at http://127.0.97.67:35295/ using document root <none> and password file <none>
I20260619 06:27:25.697058 1863 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:25.697206 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:25.697475 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:25.700107 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "2d2e72a138a14082a1c7b4fe68b60435"
format_stamp: "Formatted at 2026-06-19 06:27:25 on dist-test-slave-4t1f"
server_key: "fa7c91b87d44379f8270ab2b592011d5"
server_key_iv: "d08d369be3f7afd6ffa8b13d3d886b5f"
server_key_version: "encryptionkey@0"
I20260619 06:27:25.700812 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "2d2e72a138a14082a1c7b4fe68b60435"
format_stamp: "Formatted at 2026-06-19 06:27:25 on dist-test-slave-4t1f"
server_key: "fa7c91b87d44379f8270ab2b592011d5"
server_key_iv: "d08d369be3f7afd6ffa8b13d3d886b5f"
server_key_version: "encryptionkey@0"
I20260619 06:27:25.705523 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:27:25.708891 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:25.710150 1863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:27:25.710319 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "2d2e72a138a14082a1c7b4fe68b60435"
format_stamp: "Formatted at 2026-06-19 06:27:25 on dist-test-slave-4t1f"
server_key: "fa7c91b87d44379f8270ab2b592011d5"
server_key_iv: "d08d369be3f7afd6ffa8b13d3d886b5f"
server_key_version: "encryptionkey@0"
I20260619 06:27:25.710558 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:25.721827 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:25.725484 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:25.725736 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:25.728048 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:25.728173 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:25.728325 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:25.728416 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:27:25 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:25.751742 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.728698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:39797 (local address 127.0.97.126:45935)
0619 06:27:25.728940 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.728950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.730187 (+ 1237us) server_negotiation.cc:407] Connection header received
0619 06:27:25.732351 (+ 2164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.732356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.732457 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.732565 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:25.734512 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.735546 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.738067 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.738666 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.738786 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.744277 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:25.744287 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:25.744294 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:25.744337 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:25.747753 (+ 3416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.748989 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.748995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.748999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.749176 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.749775 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.749786 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.749789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.750086 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:25.750246 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.751299 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.751457 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:27:25.753674 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.729206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45935 (local address 127.0.97.67:39797)
0619 06:27:25.730010 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:25.730055 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:25.732089 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:25.732787 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:25.732797 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:25.733508 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:25.734330 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.734356 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.735716 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.735726 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.737852 (+ 2126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.737872 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.738931 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.738943 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.739285 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.740074 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:25.740112 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:25.743988 (+ 3876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:25.747951 (+ 3963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.747963 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.747983 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.748737 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.749296 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.749301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.749309 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.749615 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.750250 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:25.750264 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:25.750804 (+ 540us) client_negotiation.cc:769] Sending connection context
0619 06:27:25.751189 (+ 385us) client_negotiation.cc:240] Negotiation successful
0619 06:27:25.751843 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":380,"threads_started":1}
I20260619 06:27:25.771927 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40315
I20260619 06:27:25.772015 1997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40315 every 8 connection(s)
I20260619 06:27:25.773483 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:25.775619 1998 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45935
I20260619 06:27:25.775842 1998 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:25.776576 1998 heartbeater.cc:511] Master 127.0.97.126:45935 requested a full tablet report, sending...
I20260619 06:27:25.778062 1511 ts_manager.cc:194] Registered new tserver with Master: 2d2e72a138a14082a1c7b4fe68b60435 (127.0.97.67:40315)
I20260619 06:27:25.779105 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:39797
I20260619 06:27:25.781206 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.759909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34711 (local address 127.0.97.126:45935)
0619 06:27:25.760096 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.760104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.767494 (+ 7390us) server_negotiation.cc:407] Connection header received
0619 06:27:25.767811 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.767817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.767892 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.768003 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:25.768902 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.769748 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.772067 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.772530 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.772624 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.775247 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:25.775255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:25.775259 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:25.775295 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:25.778495 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.779228 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.779235 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.779240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.779422 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.779822 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.779825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.779827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.780167 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:25.780321 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.780682 (+ 361us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.780882 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:27:25.781216 1969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.760259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45935 (local address 127.0.97.67:34711)
0619 06:27:25.767362 (+ 7103us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:25.767385 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:25.767634 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:25.768035 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:25.768039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:25.768495 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:25.768760 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.768768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.769976 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.769982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.771796 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.771810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.772712 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.772718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.772910 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.773538 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:25.773560 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:25.774781 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:25.778611 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.778618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.778623 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.779094 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.779544 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:25.779549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:25.779552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:25.779711 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:25.780294 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:25.780298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:25.780453 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 06:27:25.780561 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:27:25.780828 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6875,"mutex_wait_us":34,"thread_start_us":6464,"threads_started":1}
I20260619 06:27:25.782524 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 1863
I20260619 06:27:25.782696 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:25.783164 389 external_mini_cluster.cc:1428] Setting key d056bb92576e1db5a85a8101730a3bff
I20260619 06:27:25.786078 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:27:25 dist-test-slave-4t1f krb5kdc[1467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:27:25 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:25.829751 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.816620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.0.97.126:45935)
0619 06:27:25.816842 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.816849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.817113 (+ 264us) server_negotiation.cc:407] Connection header received
0619 06:27:25.817320 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.817324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.817383 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.817488 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:25.818856 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.819580 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.820813 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.821292 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.821432 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.825111 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:25.825131 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:25.825139 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:25.825178 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:25.827556 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.828105 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.828109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.828111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.828223 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:25.828536 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:25.828539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:25.828541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:25.828767 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:25.828998 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.829261 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.829406 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:27:25.833549 1511 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:25.839913 1511 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:25.874229 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.864616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33853 (local address 127.0.0.1:42842)
0619 06:27:25.865650 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:25.865671 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:25.865771 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:25.866888 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:25.866893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:25.866920 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:25.867404 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.867414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.869288 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.869293 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.871521 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.871533 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.873301 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.873308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.873493 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.873543 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 06:27:25.873667 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 06:27:25.873831 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":3,"thread_start_us":1206,"threads_started":1}
I20260619 06:27:25.874583 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.851850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.0.97.67:40315)
0619 06:27:25.862783 (+ 10933us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.862795 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.865549 (+ 2754us) server_negotiation.cc:407] Connection header received
0619 06:27:25.865879 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.865890 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.866198 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.866396 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:25.867140 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.868731 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.871540 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.872983 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.873072 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.873327 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.873920 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.874078 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10698,"thread_start_us":12027,"threads_started":1}
I20260619 06:27:25.875413 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.864149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.0.97.66:33853)
0619 06:27:25.866388 (+ 2239us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.866396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.866419 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:27:25.866503 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.866510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.866687 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.867587 (+ 900us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:25.867756 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.869142 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.871711 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.873103 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.874763 (+ 1660us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.874940 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.875021 (+ 81us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.875165 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2086,"thread_start_us":1955,"threads_started":1}
I20260619 06:27:25.875808 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.851540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.0.97.65:45709)
0619 06:27:25.864583 (+ 13043us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:25.864594 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:25.864630 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:27:25.864745 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:25.864757 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:25.865047 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:25.868402 (+ 3355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:25.868572 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.870022 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.872438 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.874761 (+ 2323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.875135 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.875327 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:25.875387 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 06:27:25.875524 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12842,"thread_start_us":12696,"threads_started":1}
I20260619 06:27:25.875902 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.851465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:45709 (local address 127.0.0.1:60522)
0619 06:27:25.863446 (+ 11981us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:25.863503 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:25.863713 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:25.866743 (+ 3030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:25.866750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:25.866776 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:25.867296 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.867308 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.870172 (+ 2864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.870177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.872236 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.872250 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.875011 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.875018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.875211 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.875239 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:27:25.875318 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 06:27:25.875428 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11657,"mutex_wait_us":17,"thread_start_us":13522,"threads_started":1}
I20260619 06:27:25.876590 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:25.851524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40315 (local address 127.0.0.1:53932)
0619 06:27:25.865402 (+ 13878us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:25.865428 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:25.865667 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:25.866430 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:25.866451 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:25.866481 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:25.866894 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.866910 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.868906 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.868911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.871359 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:25.871372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:25.873523 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:25.873530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:25.873719 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:25.873753 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:27:25.874973 (+ 1220us) client_negotiation.cc:240] Negotiation successful
0619 06:27:25.876178 (+ 1205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13655,"mutex_wait_us":33,"thread_start_us":11506,"threads_started":1}
I20260619 06:27:25.879057 1789 tablet_service.cc:1467] Processing CreateTablet for tablet a931db65b77a46b6ace5e99f1059ea66 (DEFAULT_TABLE table=test-table [id=d18726165f244d95a91554a85b4e56b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:25.879287 1649 tablet_service.cc:1467] Processing CreateTablet for tablet a931db65b77a46b6ace5e99f1059ea66 (DEFAULT_TABLE table=test-table [id=d18726165f244d95a91554a85b4e56b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:25.879711 1928 tablet_service.cc:1467] Processing CreateTablet for tablet a931db65b77a46b6ace5e99f1059ea66 (DEFAULT_TABLE table=test-table [id=d18726165f244d95a91554a85b4e56b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:25.880435 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a931db65b77a46b6ace5e99f1059ea66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:25.880961 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a931db65b77a46b6ace5e99f1059ea66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:25.881517 1649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a931db65b77a46b6ace5e99f1059ea66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:25.892913 2017 tablet_bootstrap.cc:492] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Bootstrap starting.
I20260619 06:27:25.895699 2018 tablet_bootstrap.cc:492] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Bootstrap starting.
I20260619 06:27:25.896050 2017 tablet_bootstrap.cc:654] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:25.896973 2019 tablet_bootstrap.cc:492] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Bootstrap starting.
I20260619 06:27:25.897284 2017 log.cc:824] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:25.899014 2018 tablet_bootstrap.cc:654] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:25.900127 2018 log.cc:824] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:25.900698 2017 tablet_bootstrap.cc:492] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: No bootstrap required, opened a new log
I20260619 06:27:25.900753 2019 tablet_bootstrap.cc:654] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:25.900949 2017 ts_tablet_manager.cc:1397] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 06:27:25.901854 2019 log.cc:824] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:25.902647 2018 tablet_bootstrap.cc:492] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: No bootstrap required, opened a new log
I20260619 06:27:25.902885 2018 ts_tablet_manager.cc:1397] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:27:25.904896 2019 tablet_bootstrap.cc:492] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: No bootstrap required, opened a new log
I20260619 06:27:25.905164 2019 ts_tablet_manager.cc:1397] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260619 06:27:25.909641 2018 raft_consensus.cc:348] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:25.909945 2018 raft_consensus.cc:374] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:25.910058 2018 raft_consensus.cc:729] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2e72a138a14082a1c7b4fe68b60435, State: Initialized, Role: FOLLOWER
I20260619 06:27:25.910987 2018 consensus_queue.cc:260] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:25.911284 2017 raft_consensus.cc:348] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:25.911640 2017 raft_consensus.cc:374] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:25.911731 2017 raft_consensus.cc:729] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9166564c0cd46469dfd2a2cf18d63b7, State: Initialized, Role: FOLLOWER
I20260619 06:27:25.912424 2017 consensus_queue.cc:260] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:25.915712 2019 raft_consensus.cc:348] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:25.916157 2019 raft_consensus.cc:374] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:25.916262 2019 raft_consensus.cc:729] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3fbdbf30254da3904a53453f47d206, State: Initialized, Role: FOLLOWER
I20260619 06:27:25.916909 2019 consensus_queue.cc:260] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:25.917800 1998 heartbeater.cc:503] Master 127.0.97.126:45935 was elected leader, sending a full tablet report...
I20260619 06:27:25.918462 2018 ts_tablet_manager.cc:1428] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260619 06:27:25.919466 1859 heartbeater.cc:503] Master 127.0.97.126:45935 was elected leader, sending a full tablet report...
I20260619 06:27:25.920506 2017 ts_tablet_manager.cc:1428] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20260619 06:27:25.921917 2019 ts_tablet_manager.cc:1428] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Time spent starting tablet: real 0.017s user 0.011s sys 0.005s
W20260619 06:27:25.958397 1720 tablet.cc:2369] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:27:26.025146 1999 tablet.cc:2369] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:27:26.060550 1860 tablet.cc:2369] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:27:26.073532 2023 raft_consensus.cc:482] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:26.073900 2023 raft_consensus.cc:504] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:26.076138 2023 leader_election.cc:288] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709), d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66:33853)
I20260619 06:27:26.083762 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.077539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:38193 (local address 127.0.97.65:45709)
0619 06:27:26.077890 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:26.077896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:26.077916 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:27:26.077986 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:26.077993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:26.078089 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:26.078191 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:26.078748 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.079967 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.081837 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.082577 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.082723 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.082817 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:26.083396 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 06:27:26.083525 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 06:27:26.083799 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.076537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:45709 (local address 127.0.97.67:38193)
0619 06:27:26.077096 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:26.077120 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:26.077238 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:26.078265 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:26.078273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:26.078306 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:26.078564 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.078576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.080096 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.080101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.081676 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.081687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.082742 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.082748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.082960 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.083047 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 06:27:26.083374 (+ 327us) client_negotiation.cc:240] Negotiation successful
0619 06:27:26.083500 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364}
I20260619 06:27:26.084656 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.076855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33853 (local address 127.0.97.67:45937)
0619 06:27:26.078175 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:26.078195 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:26.078739 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:26.079248 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:26.079252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:26.079277 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:26.079586 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.079598 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.080913 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.080920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.082301 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.082313 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.083851 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.083856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.084062 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.084124 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 06:27:26.084212 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 06:27:26.084330 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1155,"thread_start_us":1457,"threads_started":1}
I20260619 06:27:26.085035 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931db65b77a46b6ace5e99f1059ea66" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3fbdbf30254da3904a53453f47d206" is_pre_election: true
I20260619 06:27:26.085539 1669 raft_consensus.cc:2454] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 0.
I20260619 06:27:26.085915 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.077582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45937 (local address 127.0.97.66:33853)
0619 06:27:26.077800 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:26.077808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:26.078273 (+ 465us) server_negotiation.cc:407] Connection header received
0619 06:27:26.078918 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:26.078923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:26.078998 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:26.079090 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:26.079758 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.080777 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.082443 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.083482 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.083599 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.083710 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:26.084235 (+ 525us) server_negotiation.cc:299] Negotiation successful
0619 06:27:26.084354 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:27:26.086196 1880 leader_election.cc:302] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3fbdbf30254da3904a53453f47d206, 2d2e72a138a14082a1c7b4fe68b60435; no voters:
I20260619 06:27:26.086547 2023 raft_consensus.cc:2781] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:26.086711 2023 raft_consensus.cc:482] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:26.086818 2023 raft_consensus.cc:3037] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.086897 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931db65b77a46b6ace5e99f1059ea66" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" is_pre_election: true
I20260619 06:27:26.087330 1814 raft_consensus.cc:2454] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 0.
I20260619 06:27:26.089226 2023 raft_consensus.cc:504] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:26.090148 2023 leader_election.cc:288] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 election: Requested vote from peers 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709), d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66:33853)
I20260619 06:27:26.090523 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931db65b77a46b6ace5e99f1059ea66" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9166564c0cd46469dfd2a2cf18d63b7"
I20260619 06:27:26.090703 1814 raft_consensus.cc:3037] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.092751 1814 raft_consensus.cc:2454] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 1.
I20260619 06:27:26.093014 1669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a931db65b77a46b6ace5e99f1059ea66" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3fbdbf30254da3904a53453f47d206"
I20260619 06:27:26.093230 1881 leader_election.cc:302] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2e72a138a14082a1c7b4fe68b60435, d9166564c0cd46469dfd2a2cf18d63b7; no voters:
I20260619 06:27:26.093259 1669 raft_consensus.cc:3037] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.093519 2023 raft_consensus.cc:2781] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:26.094007 2023 raft_consensus.cc:686] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2e72a138a14082a1c7b4fe68b60435, State: Running, Role: LEADER
I20260619 06:27:26.094426 2023 consensus_queue.cc:237] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } }
I20260619 06:27:26.095721 1669 raft_consensus.cc:2454] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 1.
I20260619 06:27:26.098884 1511 catalog_manager.cc:5495] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d2e72a138a14082a1c7b4fe68b60435 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } health_report { overall_health: UNKNOWN } } }
I20260619 06:27:26.159179 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.153183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53946 (local address 127.0.97.67:40315)
0619 06:27:26.153386 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:26.153392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:26.153529 (+ 137us) server_negotiation.cc:407] Connection header received
0619 06:27:26.153841 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:26.153845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:26.153924 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:26.154081 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:26.154435 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.155614 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.157000 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.157481 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.157619 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.158078 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:26.158593 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:26.158742 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:26.158849 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 06:27:26.158923 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:27:26.166476 1814 raft_consensus.cc:1261] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Refusing update from remote peer 2d2e72a138a14082a1c7b4fe68b60435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:26.166476 1669 raft_consensus.cc:1261] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Refusing update from remote peer 2d2e72a138a14082a1c7b4fe68b60435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:26.167603 2027 consensus_queue.cc:1035] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:26.168078 2023 consensus_queue.cc:1035] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:26.179509 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298459427487686656 to 7298459427205558272. Current Snapshot: MvccSnapshot[applied={T|T < 7298459427487686656}]
I20260619 06:27:26.183224 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298459427487686656 to 7298459427205558272. Current Snapshot: MvccSnapshot[applied={T|T < 7298459427487686656}]
I20260619 06:27:26.194847 1511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:26.195127 1511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:26.198503 1511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d: Sending DeleteTablet for 3 replicas of tablet a931db65b77a46b6ace5e99f1059ea66
I20260619 06:27:26.199492 1649 tablet_service.cc:1514] Processing DeleteTablet for tablet a931db65b77a46b6ace5e99f1059ea66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:26 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:60522
I20260619 06:27:26.199687 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet a931db65b77a46b6ace5e99f1059ea66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:26 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:42842
I20260619 06:27:26.199674 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet a931db65b77a46b6ace5e99f1059ea66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:26 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:53932
I20260619 06:27:26.200986 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:26.201761 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:26.211886 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 05e9b140403f441c86799138fc27a4da (DEFAULT_TABLE table=test-table [id=ccff9216c31e488089c08df569254472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:26.212324 2039 tablet_replica.cc:330] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: stopping tablet replica
I20260619 06:27:26.212224 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 05e9b140403f441c86799138fc27a4da (DEFAULT_TABLE table=test-table [id=ccff9216c31e488089c08df569254472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:26.212596 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e9b140403f441c86799138fc27a4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:26.212822 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e9b140403f441c86799138fc27a4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:26.214417 2041 tablet_replica.cc:330] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: stopping tablet replica
I20260619 06:27:26.214836 2041 raft_consensus.cc:2229] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:26.216361 2041 raft_consensus.cc:2258] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:26.217044 2040 tablet_replica.cc:330] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: stopping tablet replica
I20260619 06:27:26.217362 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 05e9b140403f441c86799138fc27a4da (DEFAULT_TABLE table=test-table [id=ccff9216c31e488089c08df569254472]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:26.217469 2040 raft_consensus.cc:2229] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:26.217787 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e9b140403f441c86799138fc27a4da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:26.219038 2040 raft_consensus.cc:2258] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:26.219800 2019 tablet_bootstrap.cc:492] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: Bootstrap starting.
I20260619 06:27:26.220396 2017 tablet_bootstrap.cc:492] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: Bootstrap starting.
I20260619 06:27:26.220613 2039 raft_consensus.cc:2229] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:26.221091 2039 raft_consensus.cc:2258] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:26.221589 2019 tablet_bootstrap.cc:654] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:26.222018 2017 tablet_bootstrap.cc:654] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:26.223235 2039 ts_tablet_manager.cc:1905] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:26.226508 2041 ts_tablet_manager.cc:1905] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:26.229383 2018 tablet_bootstrap.cc:492] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: Bootstrap starting.
I20260619 06:27:26.230953 2018 tablet_bootstrap.cc:654] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:26.231698 2039 ts_tablet_manager.cc:1918] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:26.231849 2039 log.cc:1196] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/a931db65b77a46b6ace5e99f1059ea66
I20260619 06:27:26.232352 2039 ts_tablet_manager.cc:1939] T a931db65b77a46b6ace5e99f1059ea66 P 2c3fbdbf30254da3904a53453f47d206: Deleting consensus metadata
I20260619 06:27:26.233646 2040 ts_tablet_manager.cc:1905] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:26.234392 1497 catalog_manager.cc:4841] TS 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709): tablet a931db65b77a46b6ace5e99f1059ea66 (table test-table [id=d18726165f244d95a91554a85b4e56b2]) successfully deleted
I20260619 06:27:26.237447 2019 tablet_bootstrap.cc:492] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: No bootstrap required, opened a new log
I20260619 06:27:26.237587 2019 ts_tablet_manager.cc:1397] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.003s
I20260619 06:27:26.238359 2019 raft_consensus.cc:348] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.238577 2019 raft_consensus.cc:374] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:26.238675 2019 raft_consensus.cc:729] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3fbdbf30254da3904a53453f47d206, State: Initialized, Role: FOLLOWER
I20260619 06:27:26.238929 2019 consensus_queue.cc:260] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.239596 2019 ts_tablet_manager.cc:1428] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:26.241861 2041 ts_tablet_manager.cc:1918] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:26.242019 2041 log.cc:1196] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/a931db65b77a46b6ace5e99f1059ea66
I20260619 06:27:26.242578 2041 ts_tablet_manager.cc:1939] T a931db65b77a46b6ace5e99f1059ea66 P d9166564c0cd46469dfd2a2cf18d63b7: Deleting consensus metadata
I20260619 06:27:26.244745 2017 tablet_bootstrap.cc:492] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: No bootstrap required, opened a new log
I20260619 06:27:26.244890 2017 ts_tablet_manager.cc:1397] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: Time spent bootstrapping tablet: real 0.025s user 0.015s sys 0.004s
I20260619 06:27:26.245110 1498 catalog_manager.cc:4841] TS d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66:33853): tablet a931db65b77a46b6ace5e99f1059ea66 (table test-table [id=d18726165f244d95a91554a85b4e56b2]) successfully deleted
I20260619 06:27:26.245407 2018 tablet_bootstrap.cc:492] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: No bootstrap required, opened a new log
I20260619 06:27:26.245553 2018 ts_tablet_manager.cc:1397] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260619 06:27:26.246205 2017 raft_consensus.cc:348] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.246394 2017 raft_consensus.cc:374] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:26.246364 2018 raft_consensus.cc:348] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.246493 2017 raft_consensus.cc:729] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9166564c0cd46469dfd2a2cf18d63b7, State: Initialized, Role: FOLLOWER
I20260619 06:27:26.246584 2018 raft_consensus.cc:374] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:26.246762 2018 raft_consensus.cc:729] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2e72a138a14082a1c7b4fe68b60435, State: Initialized, Role: FOLLOWER
I20260619 06:27:26.246745 2017 consensus_queue.cc:260] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.247351 2017 ts_tablet_manager.cc:1428] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:27:26.247645 2018 consensus_queue.cc:260] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.248225 2018 ts_tablet_manager.cc:1428] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:27:26.248942 2040 ts_tablet_manager.cc:1918] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:26.249126 2040 log.cc:1196] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/a931db65b77a46b6ace5e99f1059ea66
I20260619 06:27:26.249701 2040 ts_tablet_manager.cc:1939] T a931db65b77a46b6ace5e99f1059ea66 P 2d2e72a138a14082a1c7b4fe68b60435: Deleting consensus metadata
I20260619 06:27:26.251556 1498 catalog_manager.cc:4841] TS 2d2e72a138a14082a1c7b4fe68b60435 (127.0.97.67:40315): tablet a931db65b77a46b6ace5e99f1059ea66 (table test-table [id=d18726165f244d95a91554a85b4e56b2]) successfully deleted
I20260619 06:27:26.665174 2024 raft_consensus.cc:482] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:26.665396 2024 raft_consensus.cc:504] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.666929 2024 leader_election.cc:288] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709), 2d2e72a138a14082a1c7b4fe68b60435 (127.0.97.67:40315)
I20260619 06:27:26.685146 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.667403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44775 (local address 127.0.97.67:40315)
0619 06:27:26.679480 (+ 12077us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:26.679486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:26.679504 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:27:26.679570 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:26.679578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:26.679655 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:26.679783 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:26.680370 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.681139 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.683227 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.684115 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.684259 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.684383 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:26.684674 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 06:27:26.684796 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11945,"thread_start_us":11965,"threads_started":1}
I20260619 06:27:26.685307 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.667274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40315 (local address 127.0.97.66:44775)
0619 06:27:26.679128 (+ 11854us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:26.679151 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:26.679315 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:26.679803 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:26.679808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:26.679837 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:26.680207 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.680218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.681299 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.681306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.683041 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.683059 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.684293 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.684301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.684487 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.684526 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:27:26.684657 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 06:27:26.684796 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11655,"mutex_wait_us":37,"spinlock_wait_cycles":37248,"thread_start_us":11485,"threads_started":1}
I20260619 06:27:26.685842 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.667319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:45709 (local address 127.0.97.66:50277)
0619 06:27:26.679397 (+ 12078us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:26.679414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:26.679538 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:26.679990 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:26.679993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:26.680011 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:26.680368 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.680378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.681473 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.681477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.683240 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.683254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.684852 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.684858 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.685029 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.685096 (+ 67us) client_negotiation.cc:769] Sending connection context
0619 06:27:26.685186 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:27:26.685317 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11949,"mutex_wait_us":35,"thread_start_us":12079,"threads_started":1}
I20260619 06:27:26.686092 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.667377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50277 (local address 127.0.97.65:45709)
0619 06:27:26.679348 (+ 11971us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:26.679356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:26.679487 (+ 131us) server_negotiation.cc:407] Connection header received
0619 06:27:26.679685 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:26.679689 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:26.679796 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:26.679901 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:26.680535 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.681350 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.683413 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.684678 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.684801 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.684900 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:26.685612 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 06:27:26.685711 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11858,"thread_start_us":11727,"threads_started":1}
I20260619 06:27:26.686142 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9b140403f441c86799138fc27a4da" candidate_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2e72a138a14082a1c7b4fe68b60435" is_pre_election: true
I20260619 06:27:26.686570 1948 raft_consensus.cc:2454] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9166564c0cd46469dfd2a2cf18d63b7 in term 0.
I20260619 06:27:26.686661 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9b140403f441c86799138fc27a4da" candidate_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3fbdbf30254da3904a53453f47d206" is_pre_election: true
I20260619 06:27:26.687004 1668 raft_consensus.cc:2454] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d9166564c0cd46469dfd2a2cf18d63b7 in term 0.
I20260619 06:27:26.687196 1741 leader_election.cc:302] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2e72a138a14082a1c7b4fe68b60435, d9166564c0cd46469dfd2a2cf18d63b7; no voters:
I20260619 06:27:26.687635 2024 raft_consensus.cc:2781] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:26.687736 2024 raft_consensus.cc:482] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:26.687804 2024 raft_consensus.cc:3037] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.689787 2024 raft_consensus.cc:504] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.690313 2024 leader_election.cc:288] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [CANDIDATE]: Term 1 election: Requested vote from peers 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709), 2d2e72a138a14082a1c7b4fe68b60435 (127.0.97.67:40315)
I20260619 06:27:26.690634 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9b140403f441c86799138fc27a4da" candidate_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3fbdbf30254da3904a53453f47d206"
I20260619 06:27:26.690817 1668 raft_consensus.cc:3037] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.690755 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e9b140403f441c86799138fc27a4da" candidate_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d2e72a138a14082a1c7b4fe68b60435"
I20260619 06:27:26.690964 1948 raft_consensus.cc:3037] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.692680 1668 raft_consensus.cc:2454] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9166564c0cd46469dfd2a2cf18d63b7 in term 1.
I20260619 06:27:26.692765 1948 raft_consensus.cc:2454] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9166564c0cd46469dfd2a2cf18d63b7 in term 1.
I20260619 06:27:26.693120 1740 leader_election.cc:302] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3fbdbf30254da3904a53453f47d206, d9166564c0cd46469dfd2a2cf18d63b7; no voters:
I20260619 06:27:26.693403 2024 raft_consensus.cc:2781] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:26.694137 2024 raft_consensus.cc:686] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 LEADER]: Becoming Leader. State: Replica: d9166564c0cd46469dfd2a2cf18d63b7, State: Running, Role: LEADER
I20260619 06:27:26.694500 2024 consensus_queue.cc:237] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.697403 1512 catalog_manager.cc:5495] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } health_report { overall_health: UNKNOWN } } }
Jun 19 06:27:26 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:26.824994 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.811184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45935 (local address 127.0.0.1:38020)
0619 06:27:26.811943 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:26.811959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:26.812071 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:26.812472 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:26.812475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:26.813140 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:26.813356 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.813365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.814522 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.814525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.815531 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.815541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.816201 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.816205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.816342 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.817166 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:26.817185 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:26.819640 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:26.822626 (+ 2986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:26.822635 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:26.822640 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:26.823006 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:26.823387 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:26.823393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:26.823397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:26.823534 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:26.823981 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:26.823988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:26.824149 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 06:27:26.824304 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 06:27:26.824574 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":467,"threads_started":1}
I20260619 06:27:26.825050 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.811215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.0.97.126:45935)
0619 06:27:26.811881 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:26.811888 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:26.812032 (+ 144us) server_negotiation.cc:407] Connection header received
0619 06:27:26.812250 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:26.812258 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:26.812328 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:26.812432 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:26.813533 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.814377 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.815652 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.816054 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.816140 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.819812 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:26.819816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:26.819819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:26.819845 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:26.822525 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:26.823106 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:26.823110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:26.823112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:26.823257 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:26.823633 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:26.823640 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:26.823644 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:26.823867 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:26.824002 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:26.824386 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 06:27:26.824685 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":526,"thread_start_us":388,"threads_started":1}
I20260619 06:27:26.829595 1512 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38020:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:27:26.830585 1512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:26.836534 1648 tablet_service.cc:1467] Processing CreateTablet for tablet 0b792e2923e948fc9f954d6dd3f269d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f3a94b2e2834879ac31a8aea7745f9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:27:26.836622 1788 tablet_service.cc:1467] Processing CreateTablet for tablet 0b792e2923e948fc9f954d6dd3f269d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f3a94b2e2834879ac31a8aea7745f9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:27:26.836813 1927 tablet_service.cc:1467] Processing CreateTablet for tablet 0b792e2923e948fc9f954d6dd3f269d0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9f3a94b2e2834879ac31a8aea7745f9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:27:26.837039 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b792e2923e948fc9f954d6dd3f269d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:26.837154 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b792e2923e948fc9f954d6dd3f269d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:26.837272 1927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b792e2923e948fc9f954d6dd3f269d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:26.842556 2058 tablet_bootstrap.cc:492] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: Bootstrap starting.
I20260619 06:27:26.843966 2058 tablet_bootstrap.cc:654] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:26.844601 2059 tablet_bootstrap.cc:492] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: Bootstrap starting.
I20260619 06:27:26.846351 2060 tablet_bootstrap.cc:492] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: Bootstrap starting.
I20260619 06:27:26.846403 2059 tablet_bootstrap.cc:654] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:26.847041 2058 tablet_bootstrap.cc:492] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: No bootstrap required, opened a new log
I20260619 06:27:26.847200 2058 ts_tablet_manager.cc:1397] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260619 06:27:26.847939 2058 raft_consensus.cc:348] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.848198 2058 raft_consensus.cc:374] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:26.848284 2060 tablet_bootstrap.cc:654] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:26.848300 2058 raft_consensus.cc:729] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9166564c0cd46469dfd2a2cf18d63b7, State: Initialized, Role: FOLLOWER
I20260619 06:27:26.848934 2058 consensus_queue.cc:260] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.849521 2058 ts_tablet_manager.cc:1428] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:26.849380 2024 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:26.849651 2024 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:26.850808 2060 tablet_bootstrap.cc:492] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: No bootstrap required, opened a new log
I20260619 06:27:26.850963 2060 ts_tablet_manager.cc:1397] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:27:26.851168 2059 tablet_bootstrap.cc:492] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: No bootstrap required, opened a new log
I20260619 06:27:26.851302 2059 ts_tablet_manager.cc:1397] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260619 06:27:26.851688 2060 raft_consensus.cc:348] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.851941 2060 raft_consensus.cc:374] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:26.852036 2060 raft_consensus.cc:729] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3fbdbf30254da3904a53453f47d206, State: Initialized, Role: FOLLOWER
I20260619 06:27:26.852021 2059 raft_consensus.cc:348] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.852267 2059 raft_consensus.cc:374] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:26.852355 2059 raft_consensus.cc:729] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d2e72a138a14082a1c7b4fe68b60435, State: Initialized, Role: FOLLOWER
I20260619 06:27:26.852274 2060 consensus_queue.cc:260] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.852604 2059 consensus_queue.cc:260] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.853156 2060 ts_tablet_manager.cc:1428] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 06:27:26.852990 2064 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:26.853368 2064 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:26.853812 2059 ts_tablet_manager.cc:1428] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260619 06:27:26.853766 2065 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:26.854111 2065 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:26.932209 2065 raft_consensus.cc:482] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:26.932391 2065 raft_consensus.cc:504] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.933099 2065 leader_election.cc:288] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709), d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66:33853)
I20260619 06:27:26.933676 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b792e2923e948fc9f954d6dd3f269d0" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3fbdbf30254da3904a53453f47d206" is_pre_election: true
I20260619 06:27:26.933696 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b792e2923e948fc9f954d6dd3f269d0" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" is_pre_election: true
I20260619 06:27:26.933944 1668 raft_consensus.cc:2454] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 0.
I20260619 06:27:26.933974 1814 raft_consensus.cc:2454] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 0.
I20260619 06:27:26.934353 1881 leader_election.cc:302] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d2e72a138a14082a1c7b4fe68b60435, d9166564c0cd46469dfd2a2cf18d63b7; no voters:
I20260619 06:27:26.934612 2065 raft_consensus.cc:2781] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:26.934713 2065 raft_consensus.cc:482] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:26.934849 2065 raft_consensus.cc:3037] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.936878 2065 raft_consensus.cc:504] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.937475 2065 leader_election.cc:288] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 election: Requested vote from peers 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709), d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66:33853)
I20260619 06:27:26.937844 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b792e2923e948fc9f954d6dd3f269d0" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3fbdbf30254da3904a53453f47d206"
I20260619 06:27:26.938021 1814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b792e2923e948fc9f954d6dd3f269d0" candidate_uuid: "2d2e72a138a14082a1c7b4fe68b60435" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9166564c0cd46469dfd2a2cf18d63b7"
I20260619 06:27:26.938099 1668 raft_consensus.cc:3037] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.938232 1814 raft_consensus.cc:3037] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:26.941004 1668 raft_consensus.cc:2454] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 1.
I20260619 06:27:26.941020 1814 raft_consensus.cc:2454] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d2e72a138a14082a1c7b4fe68b60435 in term 1.
I20260619 06:27:26.941450 1881 leader_election.cc:302] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c3fbdbf30254da3904a53453f47d206, 2d2e72a138a14082a1c7b4fe68b60435, d9166564c0cd46469dfd2a2cf18d63b7; no voters:
I20260619 06:27:26.941768 2065 raft_consensus.cc:2781] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:26.942169 2065 raft_consensus.cc:686] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 LEADER]: Becoming Leader. State: Replica: 2d2e72a138a14082a1c7b4fe68b60435, State: Running, Role: LEADER
I20260619 06:27:26.942433 2065 consensus_queue.cc:237] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } }
I20260619 06:27:26.944032 2066 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: TxnStatusTablet state changed. Reason: New leader 2d2e72a138a14082a1c7b4fe68b60435. Latest consensus state: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:26.944416 2066 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:27:26.945137 2068 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:27:26.945396 2068 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:27:26.945559 1512 catalog_manager.cc:5495] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d2e72a138a14082a1c7b4fe68b60435 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } health_report { overall_health: HEALTHY } } }
I20260619 06:27:26.976872 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.971340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.0.97.67:40315)
0619 06:27:26.971555 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:26.971562 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:26.971680 (+ 118us) server_negotiation.cc:407] Connection header received
0619 06:27:26.971924 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:26.971927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:26.971996 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:26.972069 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:26.972466 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.973237 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.975052 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.975494 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.975585 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.976076 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:26.976239 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:26.976331 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:26.976483 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 06:27:26.976555 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:27:26.976919 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:26.971338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40315 (local address 127.0.0.1:53960)
0619 06:27:26.971581 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:26.971601 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:26.971761 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:26.972124 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:26.972131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:26.972144 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:26.972333 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.972345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.973356 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.973362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.974929 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:26.974943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:26.975635 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:26.975641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:26.975845 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:26.975910 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:26.976336 (+ 426us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:26.976361 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:27:26.976451 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 06:27:26.976613 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 06:27:26.980697 1668 raft_consensus.cc:1261] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Refusing update from remote peer 2d2e72a138a14082a1c7b4fe68b60435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:26.980684 1814 raft_consensus.cc:1261] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Refusing update from remote peer 2d2e72a138a14082a1c7b4fe68b60435: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:26.981240 2065 consensus_queue.cc:1035] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:26.981513 2066 consensus_queue.cc:1035] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:26.992282 2064 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: TxnStatusTablet state changed. Reason: New leader 2d2e72a138a14082a1c7b4fe68b60435. Latest consensus state: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:26.992583 2064 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:26.994001 2024 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: TxnStatusTablet state changed. Reason: New leader 2d2e72a138a14082a1c7b4fe68b60435. Latest consensus state: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:26.994297 2024 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:26.995770 2071 mvcc.cc:205] Tried to move back new op lower bound from 7298459430826196992 to 7298459430677938176. Current Snapshot: MvccSnapshot[applied={T|T < 7298459430826196992}]
I20260619 06:27:27.003594 2065 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:27.003991 2065 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:27:27.005752 2024 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:27.006050 2024 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P d9166564c0cd46469dfd2a2cf18d63b7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:27.010104 2066 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:27.010345 2066 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P 2d2e72a138a14082a1c7b4fe68b60435: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:27:27.018625 2064 tablet_replica.cc:439] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d2e72a138a14082a1c7b4fe68b60435" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 } } peers { permanent_uuid: "d9166564c0cd46469dfd2a2cf18d63b7" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33853 } } peers { permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 } } }
I20260619 06:27:27.018854 2064 tablet_replica.cc:442] T 0b792e2923e948fc9f954d6dd3f269d0 P 2c3fbdbf30254da3904a53453f47d206: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:27.025327 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.018286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42846 (local address 127.0.97.66:33853)
0619 06:27:27.019895 (+ 1609us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.019899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.019915 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:27:27.019988 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.019991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.020057 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.020226 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:27.020874 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.021637 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.023114 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.023579 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.023698 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.024185 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:27.024711 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:27.024872 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.024926 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.024995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1505,"thread_start_us":1406,"threads_started":1}
I20260619 06:27:27.041229 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.034784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40315 (local address 127.0.97.66:46581)
0619 06:27:27.035438 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:27.035453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:27.035574 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:27.035903 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:27.035906 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:27.035914 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:27.036212 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:27.036219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.038115 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.038118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:27.039134 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:27.039144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.040139 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.040150 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:27.040274 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.040319 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:27.040725 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:27.040733 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:27:27.040791 (+ 58us) client_negotiation.cc:240] Negotiation successful
0619 06:27:27.040889 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":366,"threads_started":1}
I20260619 06:27:27.041250 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.034815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46581 (local address 127.0.97.67:40315)
0619 06:27:27.034988 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.034995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.035529 (+ 534us) server_negotiation.cc:407] Connection header received
0619 06:27:27.035684 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.035689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.035775 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.035867 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:27.036344 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.037979 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.039256 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.039886 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.040002 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.040490 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:27.040611 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:27.040719 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.040838 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.040915 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:27:27.050320 1668 raft_consensus.cc:1261] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Refusing update from remote peer d9166564c0cd46469dfd2a2cf18d63b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:27.050489 1948 raft_consensus.cc:1261] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Refusing update from remote peer d9166564c0cd46469dfd2a2cf18d63b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:27.051149 2049 consensus_queue.cc:1035] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3fbdbf30254da3904a53453f47d206" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 45709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:27.051491 2024 consensus_queue.cc:1035] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d2e72a138a14082a1c7b4fe68b60435" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:27.091954 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.084790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:49551 (local address 127.0.97.66:33853)
0619 06:27:27.085015 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.085024 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.086540 (+ 1516us) server_negotiation.cc:407] Connection header received
0619 06:27:27.086697 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.086701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.086755 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.086871 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:27.087501 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.088483 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.090243 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.090646 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.090759 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.091293 (+ 534us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:27.091412 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:27.091504 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.091606 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.091677 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:27:27.092023 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.084432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33853 (local address 127.0.97.67:49551)
0619 06:27:27.086434 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:27.086454 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:27.086585 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:27.086885 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:27.086889 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:27.086904 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:27.087358 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:27.087368 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.088605 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.088612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:27.090093 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:27.090104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.090816 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.090822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:27.091014 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.091082 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:27.091513 (+ 431us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:27.091521 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:27:27.091583 (+ 62us) client_negotiation.cc:240] Negotiation successful
0619 06:27:27.091681 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1820,"thread_start_us":1386,"threads_started":1}
I20260619 06:27:27.135246 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:27.135494 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:27.139178 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14c5744cd70b4e9b836a0131c6a9a11d: Sending DeleteTablet for 3 replicas of tablet 05e9b140403f441c86799138fc27a4da
I20260619 06:27:27.139892 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e9b140403f441c86799138fc27a4da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:60522
I20260619 06:27:27.140307 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e9b140403f441c86799138fc27a4da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:42842
I20260619 06:27:27.140825 1927 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e9b140403f441c86799138fc27a4da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:27 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:53932
I20260619 06:27:27.152704 2092 tablet_replica.cc:330] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: stopping tablet replica
I20260619 06:27:27.153012 2092 raft_consensus.cc:2229] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:27.153755 2092 raft_consensus.cc:2258] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:27.154928 2092 ts_tablet_manager.cc:1905] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:27.161912 2092 ts_tablet_manager.cc:1918] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:27.162101 2092 log.cc:1196] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/05e9b140403f441c86799138fc27a4da
I20260619 06:27:27.162650 2092 ts_tablet_manager.cc:1939] T 05e9b140403f441c86799138fc27a4da P 2c3fbdbf30254da3904a53453f47d206: Deleting consensus metadata
I20260619 06:27:27.163739 1497 catalog_manager.cc:4841] TS 2c3fbdbf30254da3904a53453f47d206 (127.0.97.65:45709): tablet 05e9b140403f441c86799138fc27a4da (table test-table [id=ccff9216c31e488089c08df569254472]) successfully deleted
I20260619 06:27:27.165795 2094 tablet_replica.cc:330] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: stopping tablet replica
I20260619 06:27:27.166040 2094 raft_consensus.cc:2229] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:27.166972 2094 raft_consensus.cc:2258] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:27.167474 2093 tablet_replica.cc:330] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: stopping tablet replica
I20260619 06:27:27.168484 2094 ts_tablet_manager.cc:1905] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:27.168745 2093 raft_consensus.cc:2229] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:27.169392 2093 raft_consensus.cc:2258] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:27.174486 2093 ts_tablet_manager.cc:1905] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:27.176916 2094 ts_tablet_manager.cc:1918] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:27.177135 2094 log.cc:1196] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/05e9b140403f441c86799138fc27a4da
I20260619 06:27:27.177695 2094 ts_tablet_manager.cc:1939] T 05e9b140403f441c86799138fc27a4da P 2d2e72a138a14082a1c7b4fe68b60435: Deleting consensus metadata
I20260619 06:27:27.178764 1498 catalog_manager.cc:4841] TS 2d2e72a138a14082a1c7b4fe68b60435 (127.0.97.67:40315): tablet 05e9b140403f441c86799138fc27a4da (table test-table [id=ccff9216c31e488089c08df569254472]) successfully deleted
I20260619 06:27:27.181898 2093 ts_tablet_manager.cc:1918] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:27.182025 2093 log.cc:1196] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/05e9b140403f441c86799138fc27a4da
I20260619 06:27:27.182598 2093 ts_tablet_manager.cc:1939] T 05e9b140403f441c86799138fc27a4da P d9166564c0cd46469dfd2a2cf18d63b7: Deleting consensus metadata
I20260619 06:27:27.183693 1498 catalog_manager.cc:4841] TS d9166564c0cd46469dfd2a2cf18d63b7 (127.0.97.66:33853): tablet 05e9b140403f441c86799138fc27a4da (table test-table [id=ccff9216c31e488089c08df569254472]) successfully deleted
Jun 19 06:27:27 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:27.193015 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.166042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.0.97.65:45709)
0619 06:27:27.166641 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.166649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.173685 (+ 7036us) server_negotiation.cc:407] Connection header received
0619 06:27:27.173969 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.173975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.174056 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.174137 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:27.177317 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.178407 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.179844 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.180450 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.180574 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.186116 (+ 5542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:27.186144 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:27.186163 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:27.186201 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:27.190004 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.190806 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.190815 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.190819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.190961 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.191370 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.191376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.191379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.191798 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:27.192077 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.192367 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.192610 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 06:27:27.194324 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60526
I20260619 06:27:27.214403 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.198184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.0.97.126:45935)
0619 06:27:27.198402 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.198408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.199001 (+ 593us) server_negotiation.cc:407] Connection header received
0619 06:27:27.199144 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.199150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.199230 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.199332 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:27.200865 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.202016 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.203784 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.204463 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.204597 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.207751 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:27.207757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:27.207762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:27.207798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:27.211293 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.212025 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.212030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.212034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.212195 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.212682 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.212688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.212694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.213115 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:27.213461 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.213815 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.214024 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 06:27:27.215421 1510 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38028
I20260619 06:27:27.219075 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1583
I20260619 06:27:27.236495 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.219806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.0.97.65:45709)
0619 06:27:27.220004 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.220010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.221599 (+ 1589us) server_negotiation.cc:407] Connection header received
0619 06:27:27.221782 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.221789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.221859 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.221953 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:27.223393 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.224492 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.226273 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.226898 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.227020 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.230038 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:27.230045 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:27.230050 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:27.230086 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:27.233609 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.234342 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.234348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.234352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.234485 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.234877 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.234882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.234886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.235210 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:27.235447 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.235772 (+ 325us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.235985 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260619 06:27:27.237056 1714 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60532
W20260619 06:27:27.237639 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:27.237813 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 1583
I20260619 06:27:27.260900 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1723
Jun 19 06:27:27 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:27:27.280630 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.261797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.0.97.66:33853)
0619 06:27:27.262381 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.262389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.262416 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:27:27.262487 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.262493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.262594 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.262702 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:27.264241 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.265420 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.267295 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.267968 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.268092 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.273288 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:27.273306 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:27.273326 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:27.273365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:27.277417 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.278217 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.278226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.278230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.278397 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.278818 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.278824 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.278827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.279290 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:27.279614 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.279883 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.280088 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394}
W20260619 06:27:27.281347 1854 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42854
W20260619 06:27:27.282037 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:27.282191 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 1723
I20260619 06:27:27.300647 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1863
Jun 19 06:27:27 dist-test-slave-4t1f krb5kdc[1467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850445, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:27:27.317474 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:27.301489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.0.97.67:40315)
0619 06:27:27.301687 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:27.301693 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:27.301741 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:27:27.301805 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:27.301811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:27.301892 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:27.302046 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:27.303153 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.304344 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.305627 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:27.306341 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:27.306495 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:27.310561 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:27.310579 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:27.310617 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:27.310657 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:27.314635 (+ 3978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.315301 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.315310 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.315313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.315452 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:27.315816 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:27.315827 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:27.315831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:27.316340 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:27.316667 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:27.316814 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 06:27:27.317047 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:27:27.318300 1993 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53968
W20260619 06:27:27.319022 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1863: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:27.319201 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 1863
I20260619 06:27:27.341490 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 1480
I20260619 06:27:27.342625 1552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:27.454072 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 1480
2026-06-19T06:27:27Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5232 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:27 dist-test-slave-4t1f krb5kdc[2103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:27 dist-test-slave-4t1f krb5kdc[2103](info): set up 2 sockets
Jun 19 06:27:27 dist-test-slave-4t1f krb5kdc[2103](info): commencing operation
krb5kdc: starting...
W20260619 06:27:28.541185 389 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:28 dist-test-slave-4t1f krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:28.710712 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33339
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:29.013427 2116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:29.013739 2116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:29.013897 2116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:29.020881 2116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:29.020987 2116 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:27:29.021088 2116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:29.021210 2116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:29.021265 2116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:29.021381 2116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:29.030782 2116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33339
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:29.031970 2116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:29.033245 2116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:29.044456 2122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:29.044613 2124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:29.044689 2121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:29.045588 2116 server_base.cc:1034] running on GCE node
I20260619 06:27:29.046277 2116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:29.047529 2116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:29.048760 2116 hybrid_clock.cc:648] HybridClock initialized: now 1781850449048707 us; error 35 us; skew 500 ppm
I20260619 06:27:29.049125 2116 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:29 dist-test-slave-4t1f krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:29.052134 2116 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:29.053472 2116 webserver.cc:460] Webserver started at http://127.0.97.126:41897/ using document root <none> and password file <none>
I20260619 06:27:29.054122 2116 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:29.054244 2116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:29.054543 2116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:29.057400 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "7a79f53856514c1ca20b712950033f08"
format_stamp: "Formatted at 2026-06-19 06:27:29 on dist-test-slave-4t1f"
server_key: "7041dc6ec7ba172783af5b351f56fe1b"
server_key_iv: "66f2a299d2377fa0811fbb5fcc0e2c78"
server_key_version: "encryptionkey@0"
I20260619 06:27:29.058125 2116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "7a79f53856514c1ca20b712950033f08"
format_stamp: "Formatted at 2026-06-19 06:27:29 on dist-test-slave-4t1f"
server_key: "7041dc6ec7ba172783af5b351f56fe1b"
server_key_iv: "66f2a299d2377fa0811fbb5fcc0e2c78"
server_key_version: "encryptionkey@0"
I20260619 06:27:29.062721 2116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:27:29.065968 2131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:29.067188 2116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:27:29.067348 2116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "7a79f53856514c1ca20b712950033f08"
format_stamp: "Formatted at 2026-06-19 06:27:29 on dist-test-slave-4t1f"
server_key: "7041dc6ec7ba172783af5b351f56fe1b"
server_key_iv: "66f2a299d2377fa0811fbb5fcc0e2c78"
server_key_version: "encryptionkey@0"
I20260619 06:27:29.067548 2116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:29.091468 2116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:29.095707 2116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:29.095949 2116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:29.117290 2116 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33339
I20260619 06:27:29.117300 2192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33339 every 8 connection(s)
I20260619 06:27:29.118501 2116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:29.122010 2193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:29.125126 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 2116
I20260619 06:27:29.125329 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:29.126101 389 external_mini_cluster.cc:1428] Setting key 5a6bf644ed903d0da985711f357cd431
I20260619 06:27:29.134367 2193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Bootstrap starting.
I20260619 06:27:29.137910 2193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:29.139173 2193 log.cc:824] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:27:29 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:29.142107 2193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: No bootstrap required, opened a new log
I20260619 06:27:29.149324 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:29.128120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.97.126:33339)
0619 06:27:29.128841 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:29.128852 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:29.128890 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:27:29.130548 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:29.130567 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:29.130920 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:29.131239 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:29.132806 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.134110 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.135819 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.136428 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.136545 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:29.141245 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:29.141259 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:29.141275 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:29.141316 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:29.145188 (+ 3872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:29.145771 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:29.145780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:29.145787 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:29.145978 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:29.146319 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:29.146324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:29.146328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:29.146957 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:29.147211 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:29.147566 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 06:27:29.147992 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":353,"threads_started":1}
I20260619 06:27:29.151897 2193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79f53856514c1ca20b712950033f08" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33339 } }
I20260619 06:27:29.152283 2193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:29.152380 2193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a79f53856514c1ca20b712950033f08, State: Initialized, Role: FOLLOWER
I20260619 06:27:29.152904 2193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79f53856514c1ca20b712950033f08" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33339 } }
I20260619 06:27:29.153151 2193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:29.153293 2193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:29.153476 2193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:29.155649 2193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79f53856514c1ca20b712950033f08" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33339 } }
I20260619 06:27:29.156037 2193 leader_election.cc:302] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a79f53856514c1ca20b712950033f08; no voters:
I20260619 06:27:29.156518 2193 leader_election.cc:288] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:29.156656 2198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:29.157351 2198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [term 1 LEADER]: Becoming Leader. State: Replica: 7a79f53856514c1ca20b712950033f08, State: Running, Role: LEADER
I20260619 06:27:29.157866 2198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79f53856514c1ca20b712950033f08" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33339 } }
I20260619 06:27:29.158411 2193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:29.160838 2200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a79f53856514c1ca20b712950033f08. Latest consensus state: current_term: 1 leader_uuid: "7a79f53856514c1ca20b712950033f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79f53856514c1ca20b712950033f08" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33339 } } }
I20260619 06:27:29.161263 2200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:29.160863 2199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a79f53856514c1ca20b712950033f08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a79f53856514c1ca20b712950033f08" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33339 } } }
I20260619 06:27:29.162110 2199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:29.162670 2205 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:29.167336 2205 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:29.174853 2205 catalog_manager.cc:1347] Generated new cluster ID: fae25d32a797452c96c5906e988eac35
I20260619 06:27:29.174948 2205 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:29.182833 2205 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:29.184489 2205 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:29.196461 2205 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Generated new TSK 0
I20260619 06:27:29.197357 2205 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:29.241148 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33339
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:29.553882 2219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:29.554193 2219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:29.554369 2219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:29.561630 2219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:29.561770 2219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:29.561903 2219 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:27:29.561992 2219 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:27:29.562124 2219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:29.571825 2219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:29.573050 2219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:29.574330 2219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:29.587167 2224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:29.587122 2225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:29.587682 2227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:29.588243 2219 server_base.cc:1034] running on GCE node
I20260619 06:27:29.588930 2219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:29.589851 2219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:29.591074 2219 hybrid_clock.cc:648] HybridClock initialized: now 1781850449591013 us; error 56 us; skew 500 ppm
I20260619 06:27:29.591416 2219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:29 dist-test-slave-4t1f krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:29.594465 2219 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:29.595847 2219 webserver.cc:460] Webserver started at http://127.0.97.65:43719/ using document root <none> and password file <none>
I20260619 06:27:29.596511 2219 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:29.596654 2219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:29.597029 2219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:29.599916 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "e1459c5a175f4b08b1ec6265b6a5298e"
format_stamp: "Formatted at 2026-06-19 06:27:29 on dist-test-slave-4t1f"
server_key: "382effe070dc01ca3c80bc608873d2a5"
server_key_iv: "ed0af489b8d6f09e46c37b311032bfed"
server_key_version: "encryptionkey@0"
I20260619 06:27:29.600656 2219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "e1459c5a175f4b08b1ec6265b6a5298e"
format_stamp: "Formatted at 2026-06-19 06:27:29 on dist-test-slave-4t1f"
server_key: "382effe070dc01ca3c80bc608873d2a5"
server_key_iv: "ed0af489b8d6f09e46c37b311032bfed"
server_key_version: "encryptionkey@0"
I20260619 06:27:29.605412 2219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260619 06:27:29.608605 2234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:29.609880 2219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:29.610046 2219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "e1459c5a175f4b08b1ec6265b6a5298e"
format_stamp: "Formatted at 2026-06-19 06:27:29 on dist-test-slave-4t1f"
server_key: "382effe070dc01ca3c80bc608873d2a5"
server_key_iv: "ed0af489b8d6f09e46c37b311032bfed"
server_key_version: "encryptionkey@0"
I20260619 06:27:29.610239 2219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:29.636840 2219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:29.640651 2219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:29.640921 2219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:29.643289 2219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:29.643445 2219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:29.643584 2219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:29.643677 2219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:27:29 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:29.666092 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:29.644110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45475 (local address 127.0.97.126:33339)
0619 06:27:29.644269 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:29.644275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:29.645694 (+ 1419us) server_negotiation.cc:407] Connection header received
0619 06:27:29.648715 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:29.648720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:29.648819 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:29.648939 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:29.651564 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.652949 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.654757 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.655502 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.655654 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:29.659988 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:29.659998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:29.660004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:29.660055 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:29.663047 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:29.663854 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:29.663859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:29.663863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:29.663987 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:29.664408 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:29.664412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:29.664416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:29.664671 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:29.664820 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:29.665530 (+ 710us) server_negotiation.cc:299] Negotiation successful
0619 06:27:29.665763 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:27:29.667232 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:29.644638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33339 (local address 127.0.97.65:45475)
0619 06:27:29.645516 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:29.645561 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:29.648366 (+ 2805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:29.649264 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:29.649278 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:29.650183 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:29.651284 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:29.651319 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.653151 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.653155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:29.654628 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:29.654639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.655757 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.655763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:29.656063 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:29.656708 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:29.656727 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:29.659725 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:29.663175 (+ 3450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:29.663183 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:29.663197 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:29.663720 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:29.664092 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:29.664098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:29.664101 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:29.664277 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:29.664806 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:29.664813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:29.665155 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 06:27:29.665444 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 06:27:29.665975 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":368,"threads_started":1}
I20260619 06:27:29.694235 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:29.680020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33339 (local address 127.0.97.65:49265)
0619 06:27:29.680597 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:29.680612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:29.680772 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:29.681133 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:29.681138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:29.681881 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:29.682290 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:29.682305 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.683336 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.683341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:29.684342 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:29.684352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.685104 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.685113 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:29.685307 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:29.686063 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:29.686091 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:29.686865 (+ 774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:29.690947 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:29.690954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:29.690957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:29.691337 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:29.691767 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:29.691775 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:29.691780 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:29.691979 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:29.692791 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:29.692798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:29.693095 (+ 297us) client_negotiation.cc:769] Sending connection context
0619 06:27:29.693289 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 06:27:29.693717 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":337,"threads_started":1}
I20260619 06:27:29.694448 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:29.679978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49265 (local address 127.0.97.126:33339)
0619 06:27:29.680150 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:29.680154 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:29.680700 (+ 546us) server_negotiation.cc:407] Connection header received
0619 06:27:29.680913 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:29.680918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:29.680984 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:29.681103 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:29.682507 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.683192 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.684484 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:29.684893 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:29.685033 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:29.687058 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:29.687067 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:29.687075 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:29.687135 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:29.690822 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:29.691460 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:29.691468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:29.691474 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:29.691631 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:29.692141 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:29.692150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:29.692154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:29.692532 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:29.692723 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:29.693886 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0619 06:27:29.694097 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:27:29.699801 2219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:37957
I20260619 06:27:29.699877 2353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:37957 every 8 connection(s)
I20260619 06:27:29.700995 2219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:29.702749 2354 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33339
I20260619 06:27:29.702968 2354 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:29.703583 2354 heartbeater.cc:511] Master 127.0.97.126:33339 requested a full tablet report, sending...
I20260619 06:27:29.705396 2148 ts_manager.cc:194] Registered new tserver with Master: e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957)
I20260619 06:27:29.706759 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:45475
I20260619 06:27:29.708851 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 2219
I20260619 06:27:29.709116 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:29.709743 389 external_mini_cluster.cc:1428] Setting key 1204d5ca5af62be016aa964aa259f88f
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:27:29.746047 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33339
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:30.068186 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:30.068449 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:30.068552 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:30.078517 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:30.078653 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:30.078794 2358 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:27:30.078883 2358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:27:30.079005 2358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:30.090482 2358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:30.091670 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:30.093024 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:30.106078 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:30.106135 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:30.106144 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:30.107026 2358 server_base.cc:1034] running on GCE node
I20260619 06:27:30.107687 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:30.108572 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:30.109782 2358 hybrid_clock.cc:648] HybridClock initialized: now 1781850450109742 us; error 51 us; skew 500 ppm
I20260619 06:27:30.110090 2358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:30 dist-test-slave-4t1f krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:30.113155 2358 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:30.114459 2358 webserver.cc:460] Webserver started at http://127.0.97.66:44643/ using document root <none> and password file <none>
I20260619 06:27:30.115137 2358 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:30.115273 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:30.115588 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:30.118499 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "cbef36d34259433ca1e129407d63956e"
format_stamp: "Formatted at 2026-06-19 06:27:30 on dist-test-slave-4t1f"
server_key: "d78d5bd3e20d3449ba2b2e3b3454a29b"
server_key_iv: "fa41df9001878ea0d498fdb34bf1f630"
server_key_version: "encryptionkey@0"
I20260619 06:27:30.119251 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "cbef36d34259433ca1e129407d63956e"
format_stamp: "Formatted at 2026-06-19 06:27:30 on dist-test-slave-4t1f"
server_key: "d78d5bd3e20d3449ba2b2e3b3454a29b"
server_key_iv: "fa41df9001878ea0d498fdb34bf1f630"
server_key_version: "encryptionkey@0"
I20260619 06:27:30.124272 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:27:30.127688 2373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:30.128963 2358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:27:30.129154 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "cbef36d34259433ca1e129407d63956e"
format_stamp: "Formatted at 2026-06-19 06:27:30 on dist-test-slave-4t1f"
server_key: "d78d5bd3e20d3449ba2b2e3b3454a29b"
server_key_iv: "fa41df9001878ea0d498fdb34bf1f630"
server_key_version: "encryptionkey@0"
I20260619 06:27:30.129361 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:30.150676 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:30.154883 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:30.155105 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:30.157634 2358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:30.157758 2358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:30.157915 2358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:30.158018 2358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:27:30 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:30.184027 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.158788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33339 (local address 127.0.97.66:52979)
0619 06:27:30.159608 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.159662 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.161702 (+ 2040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.162538 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.162553 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.163225 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:30.163833 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.163849 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.165269 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.165275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.167983 (+ 2708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.168003 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.169360 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.169370 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.169749 (+ 379us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.170494 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:30.170528 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:30.173735 (+ 3207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:30.178196 (+ 4461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.178208 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.178227 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.179207 (+ 980us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.179804 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.179816 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.179822 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.180090 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.180867 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:30.180875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:30.181418 (+ 543us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.181817 (+ 399us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.182462 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":354,"threads_started":1}
I20260619 06:27:30.185097 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.158142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52979 (local address 127.0.97.126:33339)
0619 06:27:30.158328 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.158335 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.159765 (+ 1430us) server_negotiation.cc:407] Connection header received
0619 06:27:30.162010 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.162016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.162101 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.162229 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:30.164020 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.165040 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.168203 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.169043 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.169205 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.173990 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:30.173996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:30.174000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:30.174040 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:30.178013 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.179446 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.179453 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.179459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.179617 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.180337 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.180345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.180351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.180705 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:30.180869 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.184440 (+ 3571us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.184728 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:27:30.208209 2460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.192451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33339 (local address 127.0.97.66:50407)
0619 06:27:30.193406 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.193424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.193587 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.193945 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.193951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.194692 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:30.194989 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.194999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.196435 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.196444 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.198427 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.198444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.199627 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.199635 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.199960 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.200761 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:30.200794 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:30.202215 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:30.205358 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.205364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.205368 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.206001 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.206396 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.206401 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.206405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.206606 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.207369 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:30.207376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:30.207527 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.207635 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.207858 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":675,"threads_started":1}
I20260619 06:27:30.208613 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.192030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50407 (local address 127.0.97.126:33339)
0619 06:27:30.192268 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.192277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.193517 (+ 1240us) server_negotiation.cc:407] Connection header received
0619 06:27:30.193722 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.193731 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.193799 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.193906 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:30.195140 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.196254 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.198644 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.199371 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.199488 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.202419 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:30.202426 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:30.202432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:30.202473 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:30.205228 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.206152 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.206158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.206161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.206281 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.206752 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.206760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.206765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.207242 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:30.207417 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.207817 (+ 400us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.208128 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:27:30.210979 2358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:42245
I20260619 06:27:30.210999 2492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:42245 every 8 connection(s)
I20260619 06:27:30.212476 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:30.214380 2493 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33339
I20260619 06:27:30.214638 2493 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:30.215211 2493 heartbeater.cc:511] Master 127.0.97.126:33339 requested a full tablet report, sending...
I20260619 06:27:30.216572 2148 ts_manager.cc:194] Registered new tserver with Master: cbef36d34259433ca1e129407d63956e (127.0.97.66:42245)
I20260619 06:27:30.217607 2148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:52979
I20260619 06:27:30.222193 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 2358
I20260619 06:27:30.222383 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:30.222798 389 external_mini_cluster.cc:1428] Setting key fda771f9c8271e63900104111e7e88b1
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:30.298254 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33339
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:30.605440 2497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:30.605674 2497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:30.605825 2497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:30.612442 2497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:30.612524 2497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:30.612679 2497 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 06:27:30.612751 2497 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:27:30.612854 2497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:30.621731 2497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:30.622788 2497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:30.624009 2497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:30.636274 2502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:30.636332 2503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:30.636575 2505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:30.637490 2497 server_base.cc:1034] running on GCE node
I20260619 06:27:30.638090 2497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:30.638845 2497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:30.640030 2497 hybrid_clock.cc:648] HybridClock initialized: now 1781850450639980 us; error 39 us; skew 500 ppm
I20260619 06:27:30.640323 2497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:30 dist-test-slave-4t1f krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:30.643136 2497 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:30.644515 2497 webserver.cc:460] Webserver started at http://127.0.97.67:45145/ using document root <none> and password file <none>
I20260619 06:27:30.645205 2497 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:30.645349 2497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:30.645651 2497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:30.648320 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "60c6de082d174520be5b3dd2b27b0835"
format_stamp: "Formatted at 2026-06-19 06:27:30 on dist-test-slave-4t1f"
server_key: "6ffb88f38d8b56444588b8a01a5d76c3"
server_key_iv: "37043b49b7a3fd8df8c9c67b879258e7"
server_key_version: "encryptionkey@0"
I20260619 06:27:30.649056 2497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "60c6de082d174520be5b3dd2b27b0835"
format_stamp: "Formatted at 2026-06-19 06:27:30 on dist-test-slave-4t1f"
server_key: "6ffb88f38d8b56444588b8a01a5d76c3"
server_key_iv: "37043b49b7a3fd8df8c9c67b879258e7"
server_key_version: "encryptionkey@0"
I20260619 06:27:30.653949 2497 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:27:30.657191 2512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:30.658541 2497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:27:30.658704 2497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "60c6de082d174520be5b3dd2b27b0835"
format_stamp: "Formatted at 2026-06-19 06:27:30 on dist-test-slave-4t1f"
server_key: "6ffb88f38d8b56444588b8a01a5d76c3"
server_key_iv: "37043b49b7a3fd8df8c9c67b879258e7"
server_key_version: "encryptionkey@0"
I20260619 06:27:30.658885 2497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:30.681190 2497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:30.684736 2497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:30.685009 2497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:30.687721 2497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:30.687837 2497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:30.687963 2497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:30.688053 2497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:27:30 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:30.709226 2354 heartbeater.cc:503] Master 127.0.97.126:33339 was elected leader, sending a full tablet report...
I20260619 06:27:30.713828 2521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.688867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33339 (local address 127.0.97.67:36065)
0619 06:27:30.689640 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.689681 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.691557 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.692358 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.692372 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.693137 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:30.693980 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.694005 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.695779 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.695786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.697944 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.697956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.699002 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.699013 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.699372 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.700149 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:30.700188 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:30.703701 (+ 3513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:30.707766 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.707778 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.707799 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.708537 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.709169 (+ 632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.709174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.709178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.709462 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.710308 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:30.710318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:30.710860 (+ 542us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.711210 (+ 350us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.711874 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":361,"threads_started":1}
I20260619 06:27:30.714438 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.688196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36065 (local address 127.0.97.126:33339)
0619 06:27:30.688397 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.688405 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.689803 (+ 1398us) server_negotiation.cc:407] Connection header received
0619 06:27:30.691853 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.691858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.691940 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.692061 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:30.694270 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.695583 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.698095 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.698655 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.698820 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.704003 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:30.704009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:30.704014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:30.704051 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:30.707616 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.708731 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.708739 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.708746 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.708905 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.709632 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.709640 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.709645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.710050 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:30.710225 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.713331 (+ 3106us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.713624 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:27:30.730670 2497 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:36359
I20260619 06:27:30.730727 2631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:36359 every 8 connection(s)
I20260619 06:27:30.732121 2497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:30.734175 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 2497
I20260619 06:27:30.734413 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:30.734407 2632 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33339
I20260619 06:27:30.734634 2632 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:30.734936 389 external_mini_cluster.cc:1428] Setting key 45d1a2d9a7a17c6e6fa2928a30775ce9
I20260619 06:27:30.735191 2632 heartbeater.cc:511] Master 127.0.97.126:33339 requested a full tablet report, sending...
I20260619 06:27:30.737222 2147 ts_manager.cc:194] Registered new tserver with Master: 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359)
I20260619 06:27:30.738436 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:36065
I20260619 06:27:30.738756 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:27:30.740895 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.725148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33339 (local address 127.0.97.67:38993)
0619 06:27:30.725860 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.725875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.726026 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.726410 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.726415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.727265 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:30.727789 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.727808 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.729022 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.729027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.730944 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.730957 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.731774 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.731781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.731944 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.732554 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:30.732581 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:30.734319 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:30.738203 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.738210 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.738214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.738730 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.739145 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.739149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.739152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.739291 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.739878 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:30.739883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:30.740045 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.740189 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.740454 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":464,"threads_started":1}
I20260619 06:27:30.740896 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.724678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:38993 (local address 127.0.97.126:33339)
0619 06:27:30.724830 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.724834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.725960 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 06:27:30.726155 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.726161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.726252 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.726352 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:30.727962 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.728899 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.731111 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.731566 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.731660 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.734488 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:30.734493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:30.734498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:30.734532 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:30.738075 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.738872 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.738877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.738880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.739033 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.739434 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.739438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.739441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.739748 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:30.739892 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.740325 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.740534 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 19 06:27:30 dist-test-slave-4t1f krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:27:30 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:30.787868 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.774443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55830 (local address 127.0.97.126:33339)
0619 06:27:30.774692 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.774700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.774972 (+ 272us) server_negotiation.cc:407] Connection header received
0619 06:27:30.775181 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.775185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.775242 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.775346 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:30.776632 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.777433 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.778675 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.779084 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.779220 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.782834 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:30.782839 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:30.782842 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:30.782874 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:30.785328 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.785910 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.785916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.785921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.786043 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.786410 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.786416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.786419 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.786731 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:30.786966 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.787289 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.787530 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:27:30.791589 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:30.798099 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:30.832999 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.809207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36359 (local address 127.0.0.1:34066)
0619 06:27:30.810062 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.810096 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.810255 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.826478 (+ 16223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.826484 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.826510 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:30.826832 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.826843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.828638 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.828643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.830537 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.830554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.832225 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.832229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.832350 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.832383 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.832469 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.832575 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"mutex_wait_us":71,"thread_start_us":515,"threads_started":1}
I20260619 06:27:30.833412 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.810402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.0.97.66:42245)
0619 06:27:30.817726 (+ 7324us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.817739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.824130 (+ 6391us) server_negotiation.cc:407] Connection header received
0619 06:27:30.824711 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.824723 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.825097 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.825273 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:30.826118 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.827478 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.830681 (+ 3203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.832007 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.832106 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.832298 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.832985 (+ 687us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.833151 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7113,"thread_start_us":6953,"threads_started":1}
I20260619 06:27:30.833815 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.810787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32912 (local address 127.0.97.65:37957)
0619 06:27:30.824938 (+ 14151us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.824946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.824972 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:27:30.825082 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.825090 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.825319 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.825503 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:30.826323 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.827482 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.830651 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.832007 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.832118 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.832298 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.833319 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.833475 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14004,"thread_start_us":13868,"threads_started":1}
I20260619 06:27:30.833845 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.809487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.0.97.67:36359)
0619 06:27:30.825915 (+ 16428us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.825922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.825969 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:27:30.826093 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.826099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.826297 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.826475 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:30.827029 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.828488 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.830723 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.832029 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.832806 (+ 777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.833007 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.833376 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.833630 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16236,"thread_start_us":16310,"threads_started":1}
I20260619 06:27:30.834218 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.809750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42245 (local address 127.0.0.1:39808)
0619 06:27:30.823974 (+ 14224us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.824001 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.824479 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.825294 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.825301 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.825327 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:30.825767 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.825782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.827630 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.827635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.830525 (+ 2890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.830538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.832465 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.832470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.832711 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.832831 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.833678 (+ 847us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.833785 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13975,"mutex_wait_us":319,"spinlock_wait_cycles":1408,"thread_start_us":14384,"threads_started":1}
I20260619 06:27:30.834333 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.810463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37957 (local address 127.0.0.1:32912)
0619 06:27:30.824274 (+ 13811us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.824297 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.824442 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.825691 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.825701 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.825744 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:30.826160 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.826168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.827603 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.827610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.830453 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.830482 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.832444 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.832455 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.832706 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.832832 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.832914 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.833969 (+ 1055us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13657,"mutex_wait_us":241,"thread_start_us":14396,"threads_started":1}
I20260619 06:27:30.837553 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 725777024c654f6c83f2ca65d21c1bfc (DEFAULT_TABLE table=test-table [id=4fd0a8ea4c26413f8b008b199f6aefb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:30.837553 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 725777024c654f6c83f2ca65d21c1bfc (DEFAULT_TABLE table=test-table [id=4fd0a8ea4c26413f8b008b199f6aefb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:30.838948 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725777024c654f6c83f2ca65d21c1bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:30.838948 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725777024c654f6c83f2ca65d21c1bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:30.839356 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 725777024c654f6c83f2ca65d21c1bfc (DEFAULT_TABLE table=test-table [id=4fd0a8ea4c26413f8b008b199f6aefb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:30.840384 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725777024c654f6c83f2ca65d21c1bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:30.850762 2651 tablet_bootstrap.cc:492] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Bootstrap starting.
I20260619 06:27:30.852980 2652 tablet_bootstrap.cc:492] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Bootstrap starting.
I20260619 06:27:30.854648 2653 tablet_bootstrap.cc:492] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Bootstrap starting.
I20260619 06:27:30.856063 2651 tablet_bootstrap.cc:654] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:30.856184 2652 tablet_bootstrap.cc:654] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:30.857385 2652 log.cc:824] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:30.857385 2651 log.cc:824] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:30.857450 2653 tablet_bootstrap.cc:654] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:30.858616 2653 log.cc:824] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:30.860334 2652 tablet_bootstrap.cc:492] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: No bootstrap required, opened a new log
I20260619 06:27:30.860597 2652 ts_tablet_manager.cc:1397] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260619 06:27:30.860716 2651 tablet_bootstrap.cc:492] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: No bootstrap required, opened a new log
I20260619 06:27:30.860984 2651 ts_tablet_manager.cc:1397] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260619 06:27:30.861542 2653 tablet_bootstrap.cc:492] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: No bootstrap required, opened a new log
I20260619 06:27:30.861775 2653 ts_tablet_manager.cc:1397] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:27:30.868363 2652 raft_consensus.cc:348] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.868711 2652 raft_consensus.cc:374] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:30.868808 2652 raft_consensus.cc:729] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbef36d34259433ca1e129407d63956e, State: Initialized, Role: FOLLOWER
I20260619 06:27:30.869428 2652 consensus_queue.cc:260] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.870494 2493 heartbeater.cc:503] Master 127.0.97.126:33339 was elected leader, sending a full tablet report...
I20260619 06:27:30.871223 2652 ts_tablet_manager.cc:1428] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260619 06:27:30.871462 2651 raft_consensus.cc:348] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.871882 2651 raft_consensus.cc:374] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:30.871982 2651 raft_consensus.cc:729] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Initialized, Role: FOLLOWER
I20260619 06:27:30.872545 2651 consensus_queue.cc:260] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.873390 2653 raft_consensus.cc:348] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.873772 2653 raft_consensus.cc:374] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:30.873860 2653 raft_consensus.cc:729] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c6de082d174520be5b3dd2b27b0835, State: Initialized, Role: FOLLOWER
I20260619 06:27:30.874241 2651 ts_tablet_manager.cc:1428] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260619 06:27:30.874539 2653 consensus_queue.cc:260] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.876513 2632 heartbeater.cc:503] Master 127.0.97.126:33339 was elected leader, sending a full tablet report...
I20260619 06:27:30.878309 2653 ts_tablet_manager.cc:1428] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260619 06:27:30.884739 2658 raft_consensus.cc:482] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:30.884990 2658 raft_consensus.cc:504] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.886433 2658 leader_election.cc:288] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359), cbef36d34259433ca1e129407d63956e (127.0.97.66:42245)
I20260619 06:27:30.893038 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.886782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:37739 (local address 127.0.97.66:42245)
0619 06:27:30.886990 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.887000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.887527 (+ 527us) server_negotiation.cc:407] Connection header received
0619 06:27:30.887781 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.887789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.887885 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.887988 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:30.888806 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.889577 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.891365 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.892089 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.892244 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.892348 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.892656 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.892787 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:27:30.893085 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.886778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36061 (local address 127.0.97.67:36359)
0619 06:27:30.886985 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.886994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.887684 (+ 690us) server_negotiation.cc:407] Connection header received
0619 06:27:30.887807 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.887812 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.887884 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.887993 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:30.888694 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.889475 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.891301 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.892081 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.892207 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.892326 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.892698 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.892823 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:27:30.893214 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.886642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42245 (local address 127.0.97.65:37739)
0619 06:27:30.887415 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.887440 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.887594 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.888163 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.888168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.888202 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:30.888630 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.888643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.889731 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.889738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.891193 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.891205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.892376 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.892381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.892496 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.892536 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.892606 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.892745 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":51,"thread_start_us":635,"threads_started":1}
I20260619 06:27:30.893229 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.886659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36359 (local address 127.0.97.65:36061)
0619 06:27:30.887581 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.887595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.887699 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.888019 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.888024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.888051 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:30.888541 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.888554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.889619 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.889627 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.891150 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.891162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.892316 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.892323 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.892485 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.892553 (+ 68us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.892656 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.892775 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":41,"spinlock_wait_cycles":16768,"thread_start_us":364,"threads_started":1}
I20260619 06:27:30.894124 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725777024c654f6c83f2ca65d21c1bfc" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835" is_pre_election: true
I20260619 06:27:30.894562 2582 raft_consensus.cc:2454] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 0.
I20260619 06:27:30.894495 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725777024c654f6c83f2ca65d21c1bfc" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e" is_pre_election: true
I20260619 06:27:30.894907 2443 raft_consensus.cc:2454] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 0.
I20260619 06:27:30.895206 2237 leader_election.cc:302] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c6de082d174520be5b3dd2b27b0835, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:30.895541 2658 raft_consensus.cc:2781] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:30.895673 2658 raft_consensus.cc:482] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:30.895794 2658 raft_consensus.cc:3037] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:30.897931 2658 raft_consensus.cc:504] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.898478 2658 leader_election.cc:288] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 election: Requested vote from peers 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359), cbef36d34259433ca1e129407d63956e (127.0.97.66:42245)
I20260619 06:27:30.898885 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725777024c654f6c83f2ca65d21c1bfc" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e"
I20260619 06:27:30.898885 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725777024c654f6c83f2ca65d21c1bfc" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835"
I20260619 06:27:30.899045 2443 raft_consensus.cc:3037] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:30.899070 2582 raft_consensus.cc:3037] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:30.901122 2443 raft_consensus.cc:2454] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 1.
I20260619 06:27:30.901206 2582 raft_consensus.cc:2454] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 1.
I20260619 06:27:30.901552 2235 leader_election.cc:302] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbef36d34259433ca1e129407d63956e, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:30.901877 2658 raft_consensus.cc:2781] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:30.902321 2658 raft_consensus.cc:686] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 1 LEADER]: Becoming Leader. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Running, Role: LEADER
I20260619 06:27:30.902727 2658 consensus_queue.cc:237] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } }
I20260619 06:27:30.907102 2148 catalog_manager.cc:5495] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e reported cstate change: term changed from 0 to 1, leader changed from <none> to e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65). New cstate: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } health_report { overall_health: UNKNOWN } } }
W20260619 06:27:30.953472 2355 tablet.cc:2369] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:27:30.964949 2494 tablet.cc:2369] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 06:27:30 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:30.976316 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.963652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33339 (local address 127.0.0.1:55840)
0619 06:27:30.964297 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:30.964314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:30.964432 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:30.964764 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:30.964768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:30.965715 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:30.965990 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.965999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.966948 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.966951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.967925 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:30.967935 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.968580 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.968585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:30.968696 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.969394 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:30.969410 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:30.971583 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:30.974219 (+ 2636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.974227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.974232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.974649 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.974959 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:30.974962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:30.974964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:30.975106 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:30.975536 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:30.975541 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:30.975672 (+ 131us) client_negotiation.cc:769] Sending connection context
0619 06:27:30.975818 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:27:30.975995 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":398,"threads_started":1}
I20260619 06:27:30.976392 2196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:30.963569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55840 (local address 127.0.97.126:33339)
0619 06:27:30.963759 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:30.963765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:30.964394 (+ 629us) server_negotiation.cc:407] Connection header received
0619 06:27:30.964528 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:30.964531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:30.964587 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:30.964667 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:30.966132 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.966846 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.968036 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:30.968444 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:30.968536 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:30.971765 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:30.971769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:30.971772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:30.971797 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:30.974111 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.974743 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.974747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.974749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.974872 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:30.975190 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:30.975193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:30.975195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:30.975437 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:30.975536 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:30.975902 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:27:30.976098 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":55}
I20260619 06:27:30.980762 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55840:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:27:30.981709 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260619 06:27:30.983920 2633 tablet.cc:2369] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:27:30.988216 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa6b388b91f46d384b91e0fab8f06b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e37fe0cfddc845a4aa26ecd6e2a12ed2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:27:30.988036 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa6b388b91f46d384b91e0fab8f06b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e37fe0cfddc845a4aa26ecd6e2a12ed2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:27:30.988279 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa6b388b91f46d384b91e0fab8f06b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e37fe0cfddc845a4aa26ecd6e2a12ed2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:27:30.988988 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa6b388b91f46d384b91e0fab8f06b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:30.988988 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa6b388b91f46d384b91e0fab8f06b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:30.989115 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa6b388b91f46d384b91e0fab8f06b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:30.993993 2653 tablet_bootstrap.cc:492] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: Bootstrap starting.
I20260619 06:27:30.995332 2653 tablet_bootstrap.cc:654] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:30.995894 2651 tablet_bootstrap.cc:492] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: Bootstrap starting.
I20260619 06:27:30.996017 2652 tablet_bootstrap.cc:492] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: Bootstrap starting.
I20260619 06:27:30.997445 2651 tablet_bootstrap.cc:654] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:30.997911 2652 tablet_bootstrap.cc:654] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:30.998654 2653 tablet_bootstrap.cc:492] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: No bootstrap required, opened a new log
I20260619 06:27:30.998835 2653 ts_tablet_manager.cc:1397] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260619 06:27:30.999614 2653 raft_consensus.cc:348] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:30.999872 2653 raft_consensus.cc:374] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:30.999951 2651 tablet_bootstrap.cc:492] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: No bootstrap required, opened a new log
I20260619 06:27:30.999970 2653 raft_consensus.cc:729] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c6de082d174520be5b3dd2b27b0835, State: Initialized, Role: FOLLOWER
I20260619 06:27:31.000084 2651 ts_tablet_manager.cc:1397] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260619 06:27:31.000206 2653 consensus_queue.cc:260] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.000494 2652 tablet_bootstrap.cc:492] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: No bootstrap required, opened a new log
I20260619 06:27:31.000617 2652 ts_tablet_manager.cc:1397] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 06:27:31.000645 2659 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.001133 2659 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:31.000828 2651 raft_consensus.cc:348] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.001219 2651 raft_consensus.cc:374] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:31.001317 2651 raft_consensus.cc:729] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Initialized, Role: FOLLOWER
I20260619 06:27:31.001390 2652 raft_consensus.cc:348] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.001600 2653 ts_tablet_manager.cc:1428] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260619 06:27:31.001641 2652 raft_consensus.cc:374] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:31.001567 2651 consensus_queue.cc:260] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.001724 2652 raft_consensus.cc:729] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbef36d34259433ca1e129407d63956e, State: Initialized, Role: FOLLOWER
I20260619 06:27:31.001961 2652 consensus_queue.cc:260] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.002004 2658 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.002274 2658 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:31.002617 2652 ts_tablet_manager.cc:1428] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:31.002451 2657 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.002806 2657 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:31.002185 2651 ts_tablet_manager.cc:1428] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:31.205219 2658 raft_consensus.cc:482] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:31.205415 2658 raft_consensus.cc:504] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.206060 2658 leader_election.cc:288] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbef36d34259433ca1e129407d63956e (127.0.97.66:42245), 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359)
I20260619 06:27:31.206568 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa6b388b91f46d384b91e0fab8f06b2" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e" is_pre_election: true
I20260619 06:27:31.206643 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa6b388b91f46d384b91e0fab8f06b2" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835" is_pre_election: true
I20260619 06:27:31.206812 2443 raft_consensus.cc:2454] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 0.
I20260619 06:27:31.206982 2582 raft_consensus.cc:2454] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 0.
I20260619 06:27:31.207307 2235 leader_election.cc:302] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbef36d34259433ca1e129407d63956e, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:31.207634 2658 raft_consensus.cc:2781] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:31.207713 2658 raft_consensus.cc:482] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:31.207829 2658 raft_consensus.cc:3037] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:31.209890 2658 raft_consensus.cc:504] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.210541 2658 leader_election.cc:288] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 election: Requested vote from peers cbef36d34259433ca1e129407d63956e (127.0.97.66:42245), 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359)
I20260619 06:27:31.210827 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa6b388b91f46d384b91e0fab8f06b2" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835"
I20260619 06:27:31.210848 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa6b388b91f46d384b91e0fab8f06b2" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e"
I20260619 06:27:31.211023 2582 raft_consensus.cc:3037] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:31.211061 2443 raft_consensus.cc:3037] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:31.213490 2443 raft_consensus.cc:2454] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 1.
I20260619 06:27:31.213876 2582 raft_consensus.cc:2454] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 1.
I20260619 06:27:31.213914 2235 leader_election.cc:302] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbef36d34259433ca1e129407d63956e, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:31.214148 2658 raft_consensus.cc:2781] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:31.214334 2658 raft_consensus.cc:686] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 LEADER]: Becoming Leader. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Running, Role: LEADER
I20260619 06:27:31.214568 2658 consensus_queue.cc:237] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.216213 2662 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: TxnStatusTablet state changed. Reason: New leader e1459c5a175f4b08b1ec6265b6a5298e. Latest consensus state: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.216571 2662 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:27:31.217306 2674 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:27:31.217607 2674 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:27:31.218500 2146 catalog_manager.cc:5495] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e reported cstate change: term changed from 0 to 1, leader changed from <none> to e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65). New cstate: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } health_report { overall_health: UNKNOWN } } }
I20260619 06:27:31.240212 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.235073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.0.97.65:37957)
0619 06:27:31.235249 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.235255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.235347 (+ 92us) server_negotiation.cc:407] Connection header received
0619 06:27:31.235622 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.235626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.235720 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.235854 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:31.236184 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.236928 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.238194 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.238603 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.238728 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.239156 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.239579 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.239733 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.239828 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.239908 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:27:31.240237 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.234951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37957 (local address 127.0.0.1:32920)
0619 06:27:31.235249 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:31.235271 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:31.235434 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:31.235870 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:31.235875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:31.235885 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:31.236060 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.236067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.237057 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.237079 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.238055 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.238067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.238749 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.238754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.238914 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.238961 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.239698 (+ 737us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.239716 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 06:27:31.239810 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:27:31.239946 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260619 06:27:31.247630 2582 raft_consensus.cc:1261] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Refusing update from remote peer e1459c5a175f4b08b1ec6265b6a5298e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:31.247639 2443 raft_consensus.cc:1261] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Refusing update from remote peer e1459c5a175f4b08b1ec6265b6a5298e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:31.248549 2662 consensus_queue.cc:1035] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:31.248868 2658 consensus_queue.cc:1035] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:31.258997 2659 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: TxnStatusTablet state changed. Reason: New leader e1459c5a175f4b08b1ec6265b6a5298e. Latest consensus state: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.259274 2659 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:31.262142 2658 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.262490 2658 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:27:31.265523 2677 mvcc.cc:205] Tried to move back new op lower bound from 7298459448301293568 to 7298459448177299456. Current Snapshot: MvccSnapshot[applied={T|T < 7298459448301293568 or (T in {7298459448301293568})}]
I20260619 06:27:31.267253 2657 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: TxnStatusTablet state changed. Reason: New leader e1459c5a175f4b08b1ec6265b6a5298e. Latest consensus state: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.267632 2657 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:31.268584 2662 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.268831 2662 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P e1459c5a175f4b08b1ec6265b6a5298e: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:27:31.277642 2659 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.277868 2659 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P 60c6de082d174520be5b3dd2b27b0835: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:31.281661 2657 tablet_replica.cc:439] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } } }
I20260619 06:27:31.281899 2657 tablet_replica.cc:442] T 3aa6b388b91f46d384b91e0fab8f06b2 P cbef36d34259433ca1e129407d63956e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:27:31.286618 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.280757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.0.97.65:37957)
0619 06:27:31.280956 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.280965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.281336 (+ 371us) server_negotiation.cc:407] Connection header received
0619 06:27:31.281394 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.281399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.281484 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.281571 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:31.282203 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.283357 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.284678 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.285351 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.285478 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.285876 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.286025 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.286116 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.286237 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.286313 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:27:31.301985 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.296648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:43903 (local address 127.0.97.65:37957)
0619 06:27:31.296810 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.296814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.297341 (+ 527us) server_negotiation.cc:407] Connection header received
0619 06:27:31.297521 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.297526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.297594 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.297683 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:31.298124 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.298827 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.300187 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.300626 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.300759 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.301265 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.301343 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.301470 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.301607 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.301671 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:27:31.302146 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.296605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37957 (local address 127.0.97.65:43903)
0619 06:27:31.297255 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:31.297269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:31.297412 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:31.297690 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:31.297694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:31.297703 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:31.297999 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.298006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.298958 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.298963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.300053 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.300063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.300795 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.300806 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.300950 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.300992 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.301470 (+ 478us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.301480 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:27:31.301583 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:27:31.301728 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":4,"thread_start_us":353,"threads_started":1}
I20260619 06:27:31.311311 2443 raft_consensus.cc:1261] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Refusing update from remote peer e1459c5a175f4b08b1ec6265b6a5298e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:31.311410 2582 raft_consensus.cc:1261] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Refusing update from remote peer e1459c5a175f4b08b1ec6265b6a5298e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:31.311805 2658 consensus_queue.cc:1035] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:31.312196 2662 consensus_queue.cc:1035] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:31.319548 2678 mvcc.cc:205] Tried to move back new op lower bound from 7298459448566636544 to 7298459446900166656. Current Snapshot: MvccSnapshot[applied={T|T < 7298459448566636544}]
I20260619 06:27:31.323998 2679 mvcc.cc:205] Tried to move back new op lower bound from 7298459448566636544 to 7298459446900166656. Current Snapshot: MvccSnapshot[applied={T|T < 7298459448566636544}]
I20260619 06:27:31.372399 2148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:31.372722 2148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:31.376000 2148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Sending DeleteTablet for 3 replicas of tablet 725777024c654f6c83f2ca65d21c1bfc
I20260619 06:27:31.376999 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 725777024c654f6c83f2ca65d21c1bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:31 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34066
I20260619 06:27:31.377166 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 725777024c654f6c83f2ca65d21c1bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:31 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:32912
I20260619 06:27:31.377847 2693 tablet_replica.cc:330] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: stopping tablet replica
I20260619 06:27:31.378100 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 725777024c654f6c83f2ca65d21c1bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:31 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:39808
I20260619 06:27:31.378784 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:31.379552 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:31.387454 2694 tablet_replica.cc:330] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: stopping tablet replica
I20260619 06:27:31.388005 2694 raft_consensus.cc:2229] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:31.378422 2693 raft_consensus.cc:2229] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:31.389138 2693 raft_consensus.cc:2258] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:31.389216 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 49ade7907dd3403193d1c21706d017fd (DEFAULT_TABLE table=test-table [id=b6659fd794bb461fb29ba4cd543909cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:31.389691 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ade7907dd3403193d1c21706d017fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:31.390115 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 49ade7907dd3403193d1c21706d017fd (DEFAULT_TABLE table=test-table [id=b6659fd794bb461fb29ba4cd543909cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:31.390545 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ade7907dd3403193d1c21706d017fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:31.391346 2693 ts_tablet_manager.cc:1905] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:31.397663 2653 tablet_bootstrap.cc:492] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: Bootstrap starting.
I20260619 06:27:31.397914 2694 raft_consensus.cc:2258] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:31.399029 2693 ts_tablet_manager.cc:1918] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:31.399266 2693 log.cc:1196] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/725777024c654f6c83f2ca65d21c1bfc
I20260619 06:27:31.399294 2653 tablet_bootstrap.cc:654] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:31.399991 2693 ts_tablet_manager.cc:1939] T 725777024c654f6c83f2ca65d21c1bfc P 60c6de082d174520be5b3dd2b27b0835: Deleting consensus metadata
I20260619 06:27:31.402292 2694 ts_tablet_manager.cc:1905] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:31.403692 2134 catalog_manager.cc:4841] TS 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359): tablet 725777024c654f6c83f2ca65d21c1bfc (table test-table [id=4fd0a8ea4c26413f8b008b199f6aefb2]) successfully deleted
I20260619 06:27:31.405787 2651 tablet_bootstrap.cc:492] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: Bootstrap starting.
I20260619 06:27:31.407388 2651 tablet_bootstrap.cc:654] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:31.410233 2422 tablet_service.cc:1467] Processing CreateTablet for tablet 49ade7907dd3403193d1c21706d017fd (DEFAULT_TABLE table=test-table [id=b6659fd794bb461fb29ba4cd543909cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:31.410717 2422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ade7907dd3403193d1c21706d017fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:31.411579 2694 ts_tablet_manager.cc:1918] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:31.411824 2694 log.cc:1196] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/725777024c654f6c83f2ca65d21c1bfc
I20260619 06:27:31.412585 2694 ts_tablet_manager.cc:1939] T 725777024c654f6c83f2ca65d21c1bfc P e1459c5a175f4b08b1ec6265b6a5298e: Deleting consensus metadata
I20260619 06:27:31.414479 2134 catalog_manager.cc:4841] TS e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957): tablet 725777024c654f6c83f2ca65d21c1bfc (table test-table [id=4fd0a8ea4c26413f8b008b199f6aefb2]) successfully deleted
I20260619 06:27:31.417501 2652 tablet_bootstrap.cc:492] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: Bootstrap starting.
I20260619 06:27:31.418004 2695 tablet_replica.cc:330] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: stopping tablet replica
I20260619 06:27:31.418391 2695 raft_consensus.cc:2229] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:31.418787 2695 raft_consensus.cc:2258] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:31.419677 2652 tablet_bootstrap.cc:654] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:31.420817 2653 tablet_bootstrap.cc:492] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: No bootstrap required, opened a new log
I20260619 06:27:31.420993 2653 ts_tablet_manager.cc:1397] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.001s
I20260619 06:27:31.421762 2653 raft_consensus.cc:348] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.422005 2653 raft_consensus.cc:374] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:31.422093 2653 raft_consensus.cc:729] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c6de082d174520be5b3dd2b27b0835, State: Initialized, Role: FOLLOWER
I20260619 06:27:31.422345 2653 consensus_queue.cc:260] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.423429 2653 ts_tablet_manager.cc:1428] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:27:31.425175 2695 ts_tablet_manager.cc:1905] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:31.425943 2652 tablet_bootstrap.cc:492] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: No bootstrap required, opened a new log
I20260619 06:27:31.427486 2652 ts_tablet_manager.cc:1397] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260619 06:27:31.428115 2652 raft_consensus.cc:348] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.428396 2652 raft_consensus.cc:374] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:31.428475 2652 raft_consensus.cc:729] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbef36d34259433ca1e129407d63956e, State: Initialized, Role: FOLLOWER
I20260619 06:27:31.428791 2652 consensus_queue.cc:260] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.428977 2651 tablet_bootstrap.cc:492] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: No bootstrap required, opened a new log
I20260619 06:27:31.429210 2651 ts_tablet_manager.cc:1397] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.005s
I20260619 06:27:31.429986 2651 raft_consensus.cc:348] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.431231 2651 raft_consensus.cc:374] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:31.431380 2652 ts_tablet_manager.cc:1428] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260619 06:27:31.431365 2651 raft_consensus.cc:729] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Initialized, Role: FOLLOWER
I20260619 06:27:31.431790 2651 consensus_queue.cc:260] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.434216 2651 ts_tablet_manager.cc:1428] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260619 06:27:31.434643 2695 ts_tablet_manager.cc:1918] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:31.434787 2695 log.cc:1196] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/725777024c654f6c83f2ca65d21c1bfc
I20260619 06:27:31.435249 2695 ts_tablet_manager.cc:1939] T 725777024c654f6c83f2ca65d21c1bfc P cbef36d34259433ca1e129407d63956e: Deleting consensus metadata
I20260619 06:27:31.436818 2132 catalog_manager.cc:4841] TS cbef36d34259433ca1e129407d63956e (127.0.97.66:42245): tablet 725777024c654f6c83f2ca65d21c1bfc (table test-table [id=4fd0a8ea4c26413f8b008b199f6aefb2]) successfully deleted
I20260619 06:27:31.452672 2659 raft_consensus.cc:482] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:31.452910 2659 raft_consensus.cc:504] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.454617 2659 leader_election.cc:288] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957), cbef36d34259433ca1e129407d63956e (127.0.97.66:42245)
I20260619 06:27:31.462337 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.455125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:47131 (local address 127.0.97.65:37957)
0619 06:27:31.455325 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.455331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.455678 (+ 347us) server_negotiation.cc:407] Connection header received
0619 06:27:31.455979 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.455985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.456056 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.456117 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:31.456739 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.457796 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.460347 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.461388 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.461543 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.461667 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.461924 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.462045 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 06:27:31.462488 2699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.454765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37957 (local address 127.0.97.67:47131)
0619 06:27:31.455578 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:31.455598 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:31.455798 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:31.456310 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:31.456315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:31.456339 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:31.456582 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.456593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.457948 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.458002 (+ 54us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.460179 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.460193 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.461552 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.461557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.461752 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.461775 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:27:31.461909 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 06:27:31.462032 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":1,"thread_start_us":580,"threads_started":1}
I20260619 06:27:31.463359 2700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.455024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42245 (local address 127.0.97.67:55905)
0619 06:27:31.455926 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:31.455954 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:31.456086 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:31.456879 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:31.456886 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:31.456911 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:31.457264 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.457275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.458551 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.458557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.460605 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.460619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.462305 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.462312 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.462770 (+ 458us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.462806 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:27:31.462874 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:27:31.463011 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":2,"thread_start_us":609,"threads_started":1}
I20260619 06:27:31.463403 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ade7907dd3403193d1c21706d017fd" candidate_uuid: "60c6de082d174520be5b3dd2b27b0835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" is_pre_election: true
I20260619 06:27:31.463783 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.455382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55905 (local address 127.0.97.66:42245)
0619 06:27:31.456577 (+ 1195us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.456581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.456596 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:27:31.456651 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.456654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.456742 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.456845 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:31.457465 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.458410 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.460770 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.462083 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.462996 (+ 913us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.463104 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.463152 (+ 48us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.463258 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1055,"thread_start_us":1059,"threads_started":1}
I20260619 06:27:31.463828 2308 raft_consensus.cc:2454] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c6de082d174520be5b3dd2b27b0835 in term 0.
I20260619 06:27:31.464377 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ade7907dd3403193d1c21706d017fd" candidate_uuid: "60c6de082d174520be5b3dd2b27b0835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e" is_pre_election: true
I20260619 06:27:31.464432 2515 leader_election.cc:302] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c6de082d174520be5b3dd2b27b0835, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:31.464597 2443 raft_consensus.cc:2454] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60c6de082d174520be5b3dd2b27b0835 in term 0.
I20260619 06:27:31.464810 2659 raft_consensus.cc:2781] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:31.464934 2659 raft_consensus.cc:482] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:31.465090 2659 raft_consensus.cc:3037] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:31.467128 2659 raft_consensus.cc:504] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.467737 2659 leader_election.cc:288] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [CANDIDATE]: Term 1 election: Requested vote from peers e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957), cbef36d34259433ca1e129407d63956e (127.0.97.66:42245)
I20260619 06:27:31.468158 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ade7907dd3403193d1c21706d017fd" candidate_uuid: "60c6de082d174520be5b3dd2b27b0835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1459c5a175f4b08b1ec6265b6a5298e"
I20260619 06:27:31.468274 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ade7907dd3403193d1c21706d017fd" candidate_uuid: "60c6de082d174520be5b3dd2b27b0835" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e"
I20260619 06:27:31.468357 2308 raft_consensus.cc:3037] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:31.468447 2443 raft_consensus.cc:3037] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:31.471356 2443 raft_consensus.cc:2454] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c6de082d174520be5b3dd2b27b0835 in term 1.
I20260619 06:27:31.471638 2308 raft_consensus.cc:2454] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60c6de082d174520be5b3dd2b27b0835 in term 1.
I20260619 06:27:31.471886 2513 leader_election.cc:302] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c6de082d174520be5b3dd2b27b0835, cbef36d34259433ca1e129407d63956e; no voters:
I20260619 06:27:31.472771 2659 raft_consensus.cc:2781] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:31.473227 2659 raft_consensus.cc:686] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 1 LEADER]: Becoming Leader. State: Replica: 60c6de082d174520be5b3dd2b27b0835, State: Running, Role: LEADER
I20260619 06:27:31.473616 2659 consensus_queue.cc:237] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:31.477118 2146 catalog_manager.cc:5495] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "60c6de082d174520be5b3dd2b27b0835" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } health_report { overall_health: HEALTHY } } }
I20260619 06:27:31.509663 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.504314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.0.97.67:36359)
0619 06:27:31.504788 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.504794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.504822 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:27:31.504889 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.504892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.504954 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.505143 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:31.505591 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.506306 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.507566 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.507981 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.508125 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.508552 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.509037 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.509203 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.509273 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.509346 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":216,"threads_started":1}
I20260619 06:27:31.524680 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.518961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37957 (local address 127.0.97.67:37395)
0619 06:27:31.519613 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:31.519627 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:31.519754 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:31.520132 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:31.520142 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:31.520151 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:31.520389 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.520395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.521778 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.521781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.522748 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.522758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.523659 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.523663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.523765 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.523806 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.524258 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.524265 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:27:31.524339 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:27:31.524424 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":359,"threads_started":1}
I20260619 06:27:31.524750 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.519015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:37395 (local address 127.0.97.65:37957)
0619 06:27:31.519210 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.519216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.519726 (+ 510us) server_negotiation.cc:407] Connection header received
0619 06:27:31.519886 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.519892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.520006 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.520105 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:31.520529 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.521656 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.522891 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.523505 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.523630 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.523975 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.524144 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.524265 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.524384 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.524460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:27:31.533244 2443 raft_consensus.cc:1261] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Refusing update from remote peer 60c6de082d174520be5b3dd2b27b0835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:31.533447 2308 raft_consensus.cc:1261] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Refusing update from remote peer 60c6de082d174520be5b3dd2b27b0835: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:31.533914 2702 consensus_queue.cc:1035] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:31.534322 2659 consensus_queue.cc:1035] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:31.572322 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.566477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:48367 (local address 127.0.97.67:36359)
0619 06:27:31.566633 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:31.566638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:31.566656 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:27:31.566712 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:31.566717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:31.566791 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:31.566896 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:31.567271 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.568266 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.570262 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.570887 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.571037 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.571584 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.571740 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.571832 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:31.572005 (+ 173us) server_negotiation.cc:299] Negotiation successful
0619 06:27:31.572108 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:27:31.572463 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:31.565950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36359 (local address 127.0.97.65:48367)
0619 06:27:31.566121 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:31.566137 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:31.566224 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:31.566928 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:31.566932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:31.566944 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:31.567124 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.567140 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.568395 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.568402 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.570130 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:31.570146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:31.571073 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:31.571082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:31.571308 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:31.571372 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:31.571880 (+ 508us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:31.571890 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:27:31.571972 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:27:31.572154 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
W20260619 06:27:31.573406 2678 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:27:33.164340 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:33.164574 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:33.168226 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Sending DeleteTablet for 3 replicas of tablet 49ade7907dd3403193d1c21706d017fd
I20260619 06:27:33.168923 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ade7907dd3403193d1c21706d017fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:32912
I20260619 06:27:33.169128 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ade7907dd3403193d1c21706d017fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34066
I20260619 06:27:33.169200 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ade7907dd3403193d1c21706d017fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:39808
I20260619 06:27:33.180912 2737 tablet_replica.cc:330] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: stopping tablet replica
I20260619 06:27:33.181210 2737 raft_consensus.cc:2229] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:33.181865 2737 raft_consensus.cc:2258] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:33.182354 2738 tablet_replica.cc:330] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: stopping tablet replica
I20260619 06:27:33.182566 2738 raft_consensus.cc:2229] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:33.182783 2738 raft_consensus.cc:2258] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:33.182807 2737 ts_tablet_manager.cc:1905] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:33.184458 2738 ts_tablet_manager.cc:1905] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:33.187505 2737 ts_tablet_manager.cc:1918] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:27:33.187640 2737 log.cc:1196] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/49ade7907dd3403193d1c21706d017fd
I20260619 06:27:33.188108 2737 ts_tablet_manager.cc:1939] T 49ade7907dd3403193d1c21706d017fd P 60c6de082d174520be5b3dd2b27b0835: Deleting consensus metadata
I20260619 06:27:33.189114 2134 catalog_manager.cc:4841] TS 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359): tablet 49ade7907dd3403193d1c21706d017fd (table test-table [id=b6659fd794bb461fb29ba4cd543909cf]) successfully deleted
I20260619 06:27:33.189749 2736 tablet_replica.cc:330] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: stopping tablet replica
I20260619 06:27:33.189971 2736 raft_consensus.cc:2229] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:33.190224 2736 raft_consensus.cc:2258] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:33.191154 2736 ts_tablet_manager.cc:1905] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:33.194759 2738 ts_tablet_manager.cc:1918] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:27:33.194896 2738 log.cc:1196] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/49ade7907dd3403193d1c21706d017fd
I20260619 06:27:33.195348 2738 ts_tablet_manager.cc:1939] T 49ade7907dd3403193d1c21706d017fd P cbef36d34259433ca1e129407d63956e: Deleting consensus metadata
I20260619 06:27:33.195595 2736 ts_tablet_manager.cc:1918] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:27:33.195765 2736 log.cc:1196] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/49ade7907dd3403193d1c21706d017fd
I20260619 06:27:33.196235 2736 ts_tablet_manager.cc:1939] T 49ade7907dd3403193d1c21706d017fd P e1459c5a175f4b08b1ec6265b6a5298e: Deleting consensus metadata
I20260619 06:27:33.196353 2132 catalog_manager.cc:4841] TS cbef36d34259433ca1e129407d63956e (127.0.97.66:42245): tablet 49ade7907dd3403193d1c21706d017fd (table test-table [id=b6659fd794bb461fb29ba4cd543909cf]) successfully deleted
I20260619 06:27:33.197280 2134 catalog_manager.cc:4841] TS e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957): tablet 49ade7907dd3403193d1c21706d017fd (table test-table [id=b6659fd794bb461fb29ba4cd543909cf]) successfully deleted
Jun 19 06:27:33 dist-test-slave-4t1f krb5kdc[2103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:27:33 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:33.227026 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.211569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.0.97.126:33339)
0619 06:27:33.212427 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:33.212433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:33.212456 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:27:33.212509 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:33.212514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:33.212584 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:33.212697 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:33.214156 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.215304 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.216597 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.217290 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.217404 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.221424 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:33.221431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:33.221435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:33.221472 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:33.224635 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:33.225273 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:33.225278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:33.225282 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:33.225408 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:33.225713 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:33.225718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:33.225722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:33.226042 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:33.226254 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:33.226457 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 06:27:33.226660 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":585,"threads_started":1}
I20260619 06:27:33.230374 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:33.230950 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:33.237129 2284 tablet_service.cc:1467] Processing CreateTablet for tablet e046b48c855f4bb2a6f9d69e87d7ec46 (DEFAULT_TABLE table=test-table [id=ccec60f089d941ef8c0ba84c25ae6bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:33.237166 2423 tablet_service.cc:1467] Processing CreateTablet for tablet e046b48c855f4bb2a6f9d69e87d7ec46 (DEFAULT_TABLE table=test-table [id=ccec60f089d941ef8c0ba84c25ae6bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:33.237530 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e046b48c855f4bb2a6f9d69e87d7ec46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:33.237530 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e046b48c855f4bb2a6f9d69e87d7ec46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:33.237483 2562 tablet_service.cc:1467] Processing CreateTablet for tablet e046b48c855f4bb2a6f9d69e87d7ec46 (DEFAULT_TABLE table=test-table [id=ccec60f089d941ef8c0ba84c25ae6bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:33.238075 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e046b48c855f4bb2a6f9d69e87d7ec46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:33.244572 2749 tablet_bootstrap.cc:492] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: Bootstrap starting.
I20260619 06:27:33.245379 2747 tablet_bootstrap.cc:492] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: Bootstrap starting.
I20260619 06:27:33.246471 2749 tablet_bootstrap.cc:654] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:33.247561 2747 tablet_bootstrap.cc:654] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:33.248399 2748 tablet_bootstrap.cc:492] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: Bootstrap starting.
I20260619 06:27:33.249661 2748 tablet_bootstrap.cc:654] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:33.249722 2747 tablet_bootstrap.cc:492] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: No bootstrap required, opened a new log
I20260619 06:27:33.249871 2747 ts_tablet_manager.cc:1397] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 06:27:33.250641 2747 raft_consensus.cc:348] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.250916 2747 raft_consensus.cc:374] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:33.250921 2749 tablet_bootstrap.cc:492] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: No bootstrap required, opened a new log
I20260619 06:27:33.251008 2747 raft_consensus.cc:729] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c6de082d174520be5b3dd2b27b0835, State: Initialized, Role: FOLLOWER
I20260619 06:27:33.251075 2749 ts_tablet_manager.cc:1397] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 06:27:33.251283 2747 consensus_queue.cc:260] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.251849 2747 ts_tablet_manager.cc:1428] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:33.251999 2749 raft_consensus.cc:348] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.252323 2749 raft_consensus.cc:374] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:33.252436 2749 raft_consensus.cc:729] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Initialized, Role: FOLLOWER
I20260619 06:27:33.252712 2749 consensus_queue.cc:260] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.253481 2749 ts_tablet_manager.cc:1428] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260619 06:27:33.254868 2748 tablet_bootstrap.cc:492] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: No bootstrap required, opened a new log
I20260619 06:27:33.255008 2748 ts_tablet_manager.cc:1397] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 06:27:33.256091 2748 raft_consensus.cc:348] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.256260 2748 raft_consensus.cc:374] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:33.256351 2748 raft_consensus.cc:729] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbef36d34259433ca1e129407d63956e, State: Initialized, Role: FOLLOWER
I20260619 06:27:33.256562 2748 consensus_queue.cc:260] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.257834 2748 ts_tablet_manager.cc:1428] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260619 06:27:33.293956 2753 raft_consensus.cc:482] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:33.294198 2753 raft_consensus.cc:504] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.295765 2753 leader_election.cc:288] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957), 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359)
I20260619 06:27:33.303385 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.296091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36213 (local address 127.0.97.65:37957)
0619 06:27:33.297310 (+ 1219us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:33.297317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:33.297347 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:27:33.297452 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:33.297460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:33.297556 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:33.297674 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:33.298184 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.298949 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.301646 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.302412 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.302557 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.302696 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:33.302934 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:27:33.303062 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":919,"threads_started":1}
I20260619 06:27:33.303431 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.296027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37957 (local address 127.0.97.66:36213)
0619 06:27:33.296615 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:33.296632 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:33.296795 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:33.297754 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:33.297759 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:33.297788 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:33.298021 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.298030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.299113 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.299119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.301501 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.301512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.302576 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.302583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.302782 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.302805 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:27:33.302912 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:27:33.303035 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":4,"thread_start_us":328,"threads_started":1}
I20260619 06:27:33.303617 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.297098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44083 (local address 127.0.97.67:36359)
0619 06:27:33.297873 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:33.297878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:33.297904 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:27:33.297954 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:33.297957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:33.298003 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:33.298106 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:33.298636 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.299424 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.301852 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.302603 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.302753 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.302880 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:33.303145 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 06:27:33.303268 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":475,"threads_started":1}
I20260619 06:27:33.304102 2756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.296891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36359 (local address 127.0.97.66:44083)
0619 06:27:33.297615 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:33.297635 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:33.297761 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:33.298117 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:33.298121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:33.298143 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:33.298474 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.298482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.299583 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.299590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.301701 (+ 2111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.301712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.302833 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.302841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.302995 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.303017 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:27:33.303120 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:27:33.303245 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":20,"thread_start_us":402,"threads_started":1}
I20260619 06:27:33.304257 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e046b48c855f4bb2a6f9d69e87d7ec46" candidate_uuid: "cbef36d34259433ca1e129407d63956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" is_pre_election: true
I20260619 06:27:33.304533 2308 raft_consensus.cc:2454] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbef36d34259433ca1e129407d63956e in term 0.
I20260619 06:27:33.304796 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e046b48c855f4bb2a6f9d69e87d7ec46" candidate_uuid: "cbef36d34259433ca1e129407d63956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835" is_pre_election: true
I20260619 06:27:33.305070 2582 raft_consensus.cc:2454] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cbef36d34259433ca1e129407d63956e in term 0.
I20260619 06:27:33.305055 2376 leader_election.cc:302] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbef36d34259433ca1e129407d63956e, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:33.305493 2753 raft_consensus.cc:2781] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:33.305611 2753 raft_consensus.cc:482] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:33.305727 2753 raft_consensus.cc:3037] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:33.307986 2753 raft_consensus.cc:504] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.308591 2753 leader_election.cc:288] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [CANDIDATE]: Term 1 election: Requested vote from peers e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957), 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359)
I20260619 06:27:33.308974 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e046b48c855f4bb2a6f9d69e87d7ec46" candidate_uuid: "cbef36d34259433ca1e129407d63956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1459c5a175f4b08b1ec6265b6a5298e"
I20260619 06:27:33.309177 2308 raft_consensus.cc:3037] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:33.309103 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e046b48c855f4bb2a6f9d69e87d7ec46" candidate_uuid: "cbef36d34259433ca1e129407d63956e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835"
I20260619 06:27:33.309333 2582 raft_consensus.cc:3037] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:33.311146 2308 raft_consensus.cc:2454] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbef36d34259433ca1e129407d63956e in term 1.
I20260619 06:27:33.311455 2582 raft_consensus.cc:2454] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cbef36d34259433ca1e129407d63956e in term 1.
I20260619 06:27:33.311494 2376 leader_election.cc:302] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbef36d34259433ca1e129407d63956e, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:33.311738 2753 raft_consensus.cc:2781] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:33.312165 2753 raft_consensus.cc:686] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 1 LEADER]: Becoming Leader. State: Replica: cbef36d34259433ca1e129407d63956e, State: Running, Role: LEADER
I20260619 06:27:33.312536 2753 consensus_queue.cc:237] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.316514 2146 catalog_manager.cc:5495] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e reported cstate change: term changed from 0 to 1, leader changed from <none> to cbef36d34259433ca1e129407d63956e (127.0.97.66). New cstate: current_term: 1 leader_uuid: "cbef36d34259433ca1e129407d63956e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } health_report { overall_health: UNKNOWN } } }
I20260619 06:27:33.352358 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.346766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.0.97.66:42245)
0619 06:27:33.347318 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:33.347322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:33.347356 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:27:33.347425 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:33.347428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:33.347515 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:33.347676 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:33.348080 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.348870 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.350189 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.350584 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.350719 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.351159 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:33.351703 (+ 544us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:33.351872 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:33.351942 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:27:33.352029 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
I20260619 06:27:33.367002 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.362020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:53965 (local address 127.0.97.65:37957)
0619 06:27:33.362225 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:33.362230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:33.362700 (+ 470us) server_negotiation.cc:407] Connection header received
0619 06:27:33.362809 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:33.362821 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:33.362882 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:33.362988 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:33.363498 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.364345 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.365646 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.366023 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.366111 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.366487 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:33.366579 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:33.366642 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:33.366761 (+ 119us) server_negotiation.cc:299] Negotiation successful
0619 06:27:33.366825 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:27:33.367080 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.361858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37957 (local address 127.0.97.66:53965)
0619 06:27:33.362578 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:33.362594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:33.362697 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:33.363012 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:33.363019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:33.363031 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:33.363359 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.363368 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.364454 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.364458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.365533 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.365544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.366158 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.366162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.366289 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.366334 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:33.366658 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:33.366671 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 06:27:33.366727 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 06:27:33.366813 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":401,"threads_started":1}
I20260619 06:27:33.375285 2582 raft_consensus.cc:1261] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Refusing update from remote peer cbef36d34259433ca1e129407d63956e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:33.375272 2308 raft_consensus.cc:1261] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Refusing update from remote peer cbef36d34259433ca1e129407d63956e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:33.375898 2758 consensus_queue.cc:1035] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:33.376235 2753 consensus_queue.cc:1035] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:33.402216 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.395799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60871 (local address 127.0.97.66:42245)
0619 06:27:33.395958 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:33.395964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:33.397523 (+ 1559us) server_negotiation.cc:407] Connection header received
0619 06:27:33.397625 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:33.397628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:33.397699 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:33.397765 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:33.398295 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.399184 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.400538 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.401033 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.401195 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.401551 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:33.401649 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:33.401740 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:33.401845 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:27:33.401913 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:27:33.402338 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.395721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42245 (local address 127.0.97.65:60871)
0619 06:27:33.397432 (+ 1711us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:33.397448 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:33.397548 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:33.397810 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:33.397816 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:33.397830 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:27:33.398147 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.398160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.399320 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.399325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.400395 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:33.400406 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.401221 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.401227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:33.401374 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.401405 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:27:33.401740 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:27:33.401752 (+ 12us) client_negotiation.cc:769] Sending connection context
0619 06:27:33.401829 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:27:33.401951 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"thread_start_us":1491,"threads_started":1}
I20260619 06:27:33.430771 2148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:33.431044 2148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:33.434310 2148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Sending DeleteTablet for 3 replicas of tablet e046b48c855f4bb2a6f9d69e87d7ec46
I20260619 06:27:33.435098 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet e046b48c855f4bb2a6f9d69e87d7ec46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:32912
I20260619 06:27:33.435114 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet e046b48c855f4bb2a6f9d69e87d7ec46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:39808
I20260619 06:27:33.435403 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet e046b48c855f4bb2a6f9d69e87d7ec46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:33 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34066
I20260619 06:27:33.435650 2737 tablet_replica.cc:330] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: stopping tablet replica
I20260619 06:27:33.435670 2738 tablet_replica.cc:330] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: stopping tablet replica
I20260619 06:27:33.435851 2737 raft_consensus.cc:2229] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:33.435863 2738 raft_consensus.cc:2229] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:33.436028 2737 raft_consensus.cc:2258] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:33.436347 2738 raft_consensus.cc:2258] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:33.436638 2736 tablet_replica.cc:330] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: stopping tablet replica
I20260619 06:27:33.436823 2736 raft_consensus.cc:2229] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:33.436992 2736 raft_consensus.cc:2258] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:33.437482 2737 ts_tablet_manager.cc:1905] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:33.438108 2736 ts_tablet_manager.cc:1905] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:33.438956 2738 ts_tablet_manager.cc:1905] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:33.443862 2737 ts_tablet_manager.cc:1918] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:33.444015 2737 log.cc:1196] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/e046b48c855f4bb2a6f9d69e87d7ec46
I20260619 06:27:33.444161 2736 ts_tablet_manager.cc:1918] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:33.444301 2736 log.cc:1196] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/e046b48c855f4bb2a6f9d69e87d7ec46
I20260619 06:27:33.444450 2737 ts_tablet_manager.cc:1939] T e046b48c855f4bb2a6f9d69e87d7ec46 P 60c6de082d174520be5b3dd2b27b0835: Deleting consensus metadata
I20260619 06:27:33.444429 2148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 06:27:33.444700 2736 ts_tablet_manager.cc:1939] T e046b48c855f4bb2a6f9d69e87d7ec46 P e1459c5a175f4b08b1ec6265b6a5298e: Deleting consensus metadata
W20260619 06:27:33.445168 2148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:33.445443 2134 catalog_manager.cc:4841] TS 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359): tablet e046b48c855f4bb2a6f9d69e87d7ec46 (table test-table [id=ccec60f089d941ef8c0ba84c25ae6bd5]) successfully deleted
I20260619 06:27:33.445919 2134 catalog_manager.cc:4841] TS e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957): tablet e046b48c855f4bb2a6f9d69e87d7ec46 (table test-table [id=ccec60f089d941ef8c0ba84c25ae6bd5]) successfully deleted
I20260619 06:27:33.449698 2738 ts_tablet_manager.cc:1918] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:27:33.449870 2738 log.cc:1196] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/e046b48c855f4bb2a6f9d69e87d7ec46
I20260619 06:27:33.450304 2738 ts_tablet_manager.cc:1939] T e046b48c855f4bb2a6f9d69e87d7ec46 P cbef36d34259433ca1e129407d63956e: Deleting consensus metadata
I20260619 06:27:33.451442 2132 catalog_manager.cc:4841] TS cbef36d34259433ca1e129407d63956e (127.0.97.66:42245): tablet e046b48c855f4bb2a6f9d69e87d7ec46 (table test-table [id=ccec60f089d941ef8c0ba84c25ae6bd5]) successfully deleted
I20260619 06:27:33.451742 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc73d16a20249c59eaafc9f531efeb6 (DEFAULT_TABLE table=test-table [id=b4c6c26b047c4541b1c263d9f4ffbf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:33.451767 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc73d16a20249c59eaafc9f531efeb6 (DEFAULT_TABLE table=test-table [id=b4c6c26b047c4541b1c263d9f4ffbf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:33.452162 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc73d16a20249c59eaafc9f531efeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:33.452167 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc73d16a20249c59eaafc9f531efeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:33.452605 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 6cc73d16a20249c59eaafc9f531efeb6 (DEFAULT_TABLE table=test-table [id=b4c6c26b047c4541b1c263d9f4ffbf03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:33.452996 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cc73d16a20249c59eaafc9f531efeb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:33.458237 2749 tablet_bootstrap.cc:492] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: Bootstrap starting.
I20260619 06:27:33.458668 2748 tablet_bootstrap.cc:492] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: Bootstrap starting.
I20260619 06:27:33.459371 2747 tablet_bootstrap.cc:492] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: Bootstrap starting.
I20260619 06:27:33.460253 2749 tablet_bootstrap.cc:654] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:33.460646 2748 tablet_bootstrap.cc:654] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:33.461198 2747 tablet_bootstrap.cc:654] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:33.471071 2748 tablet_bootstrap.cc:492] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: No bootstrap required, opened a new log
I20260619 06:27:33.471315 2748 ts_tablet_manager.cc:1397] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260619 06:27:33.471936 2749 tablet_bootstrap.cc:492] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: No bootstrap required, opened a new log
I20260619 06:27:33.472097 2749 ts_tablet_manager.cc:1397] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260619 06:27:33.472074 2748 raft_consensus.cc:348] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.472332 2748 raft_consensus.cc:374] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:33.472415 2748 raft_consensus.cc:729] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbef36d34259433ca1e129407d63956e, State: Initialized, Role: FOLLOWER
I20260619 06:27:33.472687 2748 consensus_queue.cc:260] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.472813 2749 raft_consensus.cc:348] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.473040 2749 raft_consensus.cc:374] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:33.473140 2749 raft_consensus.cc:729] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Initialized, Role: FOLLOWER
I20260619 06:27:33.473389 2748 ts_tablet_manager.cc:1428] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:33.473387 2749 consensus_queue.cc:260] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.473883 2749 ts_tablet_manager.cc:1428] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:27:33.476014 2747 tablet_bootstrap.cc:492] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: No bootstrap required, opened a new log
I20260619 06:27:33.476191 2747 ts_tablet_manager.cc:1397] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20260619 06:27:33.476914 2747 raft_consensus.cc:348] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.477190 2747 raft_consensus.cc:374] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:33.477262 2747 raft_consensus.cc:729] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c6de082d174520be5b3dd2b27b0835, State: Initialized, Role: FOLLOWER
I20260619 06:27:33.477494 2747 consensus_queue.cc:260] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.478545 2747 ts_tablet_manager.cc:1428] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:33.531874 2691 raft_consensus.cc:482] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:33.532073 2691 raft_consensus.cc:504] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.532720 2691 leader_election.cc:288] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cbef36d34259433ca1e129407d63956e (127.0.97.66:42245), 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359)
I20260619 06:27:33.533313 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc73d16a20249c59eaafc9f531efeb6" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e" is_pre_election: true
I20260619 06:27:33.533308 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc73d16a20249c59eaafc9f531efeb6" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835" is_pre_election: true
I20260619 06:27:33.533553 2582 raft_consensus.cc:2454] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 0.
I20260619 06:27:33.533560 2443 raft_consensus.cc:2454] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 0.
I20260619 06:27:33.533983 2237 leader_election.cc:302] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60c6de082d174520be5b3dd2b27b0835, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:33.534298 2691 raft_consensus.cc:2781] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:33.534422 2691 raft_consensus.cc:482] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:33.534516 2691 raft_consensus.cc:3037] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:33.536441 2691 raft_consensus.cc:504] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.536995 2691 leader_election.cc:288] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 election: Requested vote from peers cbef36d34259433ca1e129407d63956e (127.0.97.66:42245), 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359)
I20260619 06:27:33.537353 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc73d16a20249c59eaafc9f531efeb6" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cbef36d34259433ca1e129407d63956e"
I20260619 06:27:33.537487 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cc73d16a20249c59eaafc9f531efeb6" candidate_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60c6de082d174520be5b3dd2b27b0835"
I20260619 06:27:33.537546 2443 raft_consensus.cc:3037] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:33.537662 2582 raft_consensus.cc:3037] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:33.540822 2582 raft_consensus.cc:2454] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 1.
I20260619 06:27:33.540828 2443 raft_consensus.cc:2454] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1459c5a175f4b08b1ec6265b6a5298e in term 1.
I20260619 06:27:33.541239 2235 leader_election.cc:302] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cbef36d34259433ca1e129407d63956e, e1459c5a175f4b08b1ec6265b6a5298e; no voters:
I20260619 06:27:33.541510 2691 raft_consensus.cc:2781] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:33.541652 2691 raft_consensus.cc:686] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 LEADER]: Becoming Leader. State: Replica: e1459c5a175f4b08b1ec6265b6a5298e, State: Running, Role: LEADER
I20260619 06:27:33.541898 2691 consensus_queue.cc:237] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } }
I20260619 06:27:33.545809 2146 catalog_manager.cc:5495] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e reported cstate change: term changed from 0 to 1, leader changed from <none> to e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65). New cstate: current_term: 1 leader_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1459c5a175f4b08b1ec6265b6a5298e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 } health_report { overall_health: UNKNOWN } } }
I20260619 06:27:33.579020 2754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:33.573506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.0.97.65:37957)
0619 06:27:33.573693 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:33.573698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:33.573860 (+ 162us) server_negotiation.cc:407] Connection header received
0619 06:27:33.573979 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:33.573983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:33.574057 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:33.574148 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:33.574458 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.575598 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.576810 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:33.577912 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:33.578045 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:33.578376 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:33.578500 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:33.578595 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:33.578680 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 06:27:33.578754 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:27:33.588330 2582 raft_consensus.cc:1261] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Refusing update from remote peer e1459c5a175f4b08b1ec6265b6a5298e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:33.588342 2443 raft_consensus.cc:1261] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Refusing update from remote peer e1459c5a175f4b08b1ec6265b6a5298e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:33.588991 2731 consensus_queue.cc:1035] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Connected to new peer: Peer: permanent_uuid: "60c6de082d174520be5b3dd2b27b0835" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:33.589285 2691 consensus_queue.cc:1035] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cbef36d34259433ca1e129407d63956e" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:27:33.612234 2713 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 06:27:35.188377 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:35.188599 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55856:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:35.191896 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a79f53856514c1ca20b712950033f08: Sending DeleteTablet for 3 replicas of tablet 6cc73d16a20249c59eaafc9f531efeb6
I20260619 06:27:35.192584 2283 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc73d16a20249c59eaafc9f531efeb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:35 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:32912
I20260619 06:27:35.192746 2423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc73d16a20249c59eaafc9f531efeb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:35 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:39808
I20260619 06:27:35.192880 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cc73d16a20249c59eaafc9f531efeb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:35 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34066
I20260619 06:27:35.193395 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2219
I20260619 06:27:35.203042 2799 tablet_replica.cc:330] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: stopping tablet replica
I20260619 06:27:35.203347 2799 raft_consensus.cc:2229] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:35.204665 2799 raft_consensus.cc:2258] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:35.206257 2799 ts_tablet_manager.cc:1905] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:35.213423 2797 tablet_replica.cc:330] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: stopping tablet replica
I20260619 06:27:35.213619 2797 raft_consensus.cc:2229] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:35.213924 2797 raft_consensus.cc:2258] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:35.214884 2798 tablet_replica.cc:330] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: stopping tablet replica
I20260619 06:27:35.215106 2798 raft_consensus.cc:2229] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:35.215376 2798 raft_consensus.cc:2258] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:35.215410 2797 ts_tablet_manager.cc:1905] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:35.216871 2798 ts_tablet_manager.cc:1905] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:35.218582 2799 ts_tablet_manager.cc:1918] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:27:35.218771 2799 log.cc:1196] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/6cc73d16a20249c59eaafc9f531efeb6
I20260619 06:27:35.219242 2799 ts_tablet_manager.cc:1939] T 6cc73d16a20249c59eaafc9f531efeb6 P 60c6de082d174520be5b3dd2b27b0835: Deleting consensus metadata
I20260619 06:27:35.220242 2134 catalog_manager.cc:4841] TS 60c6de082d174520be5b3dd2b27b0835 (127.0.97.67:36359): tablet 6cc73d16a20249c59eaafc9f531efeb6 (table test-table [id=b4c6c26b047c4541b1c263d9f4ffbf03]) successfully deleted
I20260619 06:27:35.221251 2797 ts_tablet_manager.cc:1918] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:27:35.221450 2797 log.cc:1196] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/6cc73d16a20249c59eaafc9f531efeb6
I20260619 06:27:35.221936 2797 ts_tablet_manager.cc:1939] T 6cc73d16a20249c59eaafc9f531efeb6 P e1459c5a175f4b08b1ec6265b6a5298e: Deleting consensus metadata
I20260619 06:27:35.223022 2134 catalog_manager.cc:4841] TS e1459c5a175f4b08b1ec6265b6a5298e (127.0.97.65:37957): tablet 6cc73d16a20249c59eaafc9f531efeb6 (table test-table [id=b4c6c26b047c4541b1c263d9f4ffbf03]) successfully deleted
I20260619 06:27:35.223501 2798 ts_tablet_manager.cc:1918] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 06:27:35.223698 2798 log.cc:1196] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/6cc73d16a20249c59eaafc9f531efeb6
I20260619 06:27:35.224141 2798 ts_tablet_manager.cc:1939] T 6cc73d16a20249c59eaafc9f531efeb6 P cbef36d34259433ca1e129407d63956e: Deleting consensus metadata
Jun 19 06:27:35 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:35.225055 2132 catalog_manager.cc:4841] TS cbef36d34259433ca1e129407d63956e (127.0.97.66:42245): tablet 6cc73d16a20249c59eaafc9f531efeb6 (table test-table [id=b4c6c26b047c4541b1c263d9f4ffbf03]) successfully deleted
I20260619 06:27:35.230885 2801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:35.195193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32962 (local address 127.0.97.65:37957)
0619 06:27:35.203955 (+ 8762us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:35.203961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:35.203987 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:27:35.204047 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:35.204054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:35.204162 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:35.204292 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:35.206278 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:35.207888 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:35.210438 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:35.215119 (+ 4681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:35.215246 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:35.225225 (+ 9979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:35.225243 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:35.225263 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:35.225307 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:35.228408 (+ 3101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:35.229028 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:35.229035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:35.229037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:35.229185 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:35.229522 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:35.229526 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:35.229530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:35.229839 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:35.230110 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:35.230357 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 06:27:35.230581 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8601,"thread_start_us":8455,"threads_started":1}
W20260619 06:27:35.231513 2349 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32962
W20260619 06:27:35.232085 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2219: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:35.232280 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 2219
I20260619 06:27:35.262012 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2358
Jun 19 06:27:35 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:27:35.278823 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:35.262925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39826 (local address 127.0.97.66:42245)
0619 06:27:35.263475 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:35.263480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:35.263495 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:27:35.263552 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:35.263555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:35.263609 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:35.263741 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:35.265547 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:35.266299 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:35.268503 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:35.268904 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:35.269014 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:35.273479 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:35.273492 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:35.273507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:35.273545 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:35.276306 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:35.276868 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:35.276874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:35.276876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:35.277009 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:35.277506 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:35.277509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:35.277511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:35.277822 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:35.278062 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:35.278302 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 06:27:35.278458 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":335,"threads_started":1}
W20260619 06:27:35.279445 2488 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39826
W20260619 06:27:35.280105 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:35.280293 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 2358
I20260619 06:27:35.301600 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2497
Jun 19 06:27:35 dist-test-slave-4t1f krb5kdc[2103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:27:35.318001 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:35.302341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34082 (local address 127.0.97.67:36359)
0619 06:27:35.302846 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:35.302850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:35.302866 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:27:35.302919 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:35.302923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:35.302982 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:35.303076 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:35.306118 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:35.306851 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:35.308030 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:35.308385 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:35.308477 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:35.312632 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:35.312647 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:35.312663 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:35.312695 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:35.315359 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:35.316011 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:35.316018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:35.316020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:35.316126 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:35.316422 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:35.316425 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:35.316427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:35.316810 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:35.317144 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:35.317434 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 06:27:35.317675 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260619 06:27:35.318624 2627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34082
W20260619 06:27:35.319245 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2497: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:35.319391 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 2497
I20260619 06:27:35.341523 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2116
I20260619 06:27:35.342595 2188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:35.457698 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 2116
2026-06-19T06:27:35Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8004 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:35 dist-test-slave-4t1f krb5kdc[2807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:35 dist-test-slave-4t1f krb5kdc[2807](info): set up 2 sockets
Jun 19 06:27:35 dist-test-slave-4t1f krb5kdc[2807](info): commencing operation
krb5kdc: starting...
W20260619 06:27:36.545331 389 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:36 dist-test-slave-4t1f krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:36.718062 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43967
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43967
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:37.020982 2820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:37.021415 2820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:37.021591 2820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:37.028616 2820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:37.028759 2820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:37.028870 2820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:37.028977 2820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:37.029044 2820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:37.038432 2820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:37.039721 2820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:37.040992 2820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:37.052561 2828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:37.052716 2826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:37.052716 2825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:37.052857 2820 server_base.cc:1034] running on GCE node
I20260619 06:27:37.053875 2820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:37.055099 2820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:37.056326 2820 hybrid_clock.cc:648] HybridClock initialized: now 1781850457056286 us; error 46 us; skew 500 ppm
I20260619 06:27:37.056660 2820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:37 dist-test-slave-4t1f krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:37.059584 2820 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:37.060932 2820 webserver.cc:460] Webserver started at http://127.0.97.126:46053/ using document root <none> and password file <none>
I20260619 06:27:37.061604 2820 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:37.061755 2820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:37.062054 2820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:37.064785 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "9f657fdaacc946a5923777acbdc7f8d2"
format_stamp: "Formatted at 2026-06-19 06:27:37 on dist-test-slave-4t1f"
server_key: "979c2b65031d1376fe55fa84c37f61a5"
server_key_iv: "5c8ec70f202b48cb5097ab09bf0dbac9"
server_key_version: "encryptionkey@0"
I20260619 06:27:37.065526 2820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "9f657fdaacc946a5923777acbdc7f8d2"
format_stamp: "Formatted at 2026-06-19 06:27:37 on dist-test-slave-4t1f"
server_key: "979c2b65031d1376fe55fa84c37f61a5"
server_key_iv: "5c8ec70f202b48cb5097ab09bf0dbac9"
server_key_version: "encryptionkey@0"
I20260619 06:27:37.070483 2820 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:27:37.074019 2835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:37.075762 2820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:27:37.075915 2820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "9f657fdaacc946a5923777acbdc7f8d2"
format_stamp: "Formatted at 2026-06-19 06:27:37 on dist-test-slave-4t1f"
server_key: "979c2b65031d1376fe55fa84c37f61a5"
server_key_iv: "5c8ec70f202b48cb5097ab09bf0dbac9"
server_key_version: "encryptionkey@0"
I20260619 06:27:37.076139 2820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:37.094652 2820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:37.098340 2820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:37.098551 2820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:37.116124 2820 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43967
I20260619 06:27:37.116096 2886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43967 every 8 connection(s)
I20260619 06:27:37.117522 2820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:37.121044 2887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:37.123184 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 2820
I20260619 06:27:37.123381 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:37.124131 389 external_mini_cluster.cc:1428] Setting key bdb6014f2937395cd47fd0aee9554b8f
I20260619 06:27:37.133450 2887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2: Bootstrap starting.
I20260619 06:27:37.137106 2887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2: Neither blocks nor log segments found. Creating new log.
Jun 19 06:27:37 dist-test-slave-4t1f krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:37.138216 2887 log.cc:824] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:37.140903 2887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2: No bootstrap required, opened a new log
I20260619 06:27:37.145980 2890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:37.126124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.0.97.126:43967)
0619 06:27:37.126903 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:37.126914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:37.126951 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:27:37.128575 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:37.128594 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:37.128957 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:37.129308 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:37.130989 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:37.132294 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:37.134036 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:37.134656 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:37.134813 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:37.139180 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:37.139195 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:37.139213 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:37.139269 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:37.142308 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:37.143054 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:37.143059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:37.143064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:37.143234 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:37.143609 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:37.143612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:37.143614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:37.144073 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:37.144292 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:37.144671 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 06:27:37.144975 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":375,"threads_started":1}
I20260619 06:27:37.149737 2887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f657fdaacc946a5923777acbdc7f8d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43967 } }
I20260619 06:27:37.150079 2887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:37.150180 2887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f657fdaacc946a5923777acbdc7f8d2, State: Initialized, Role: FOLLOWER
I20260619 06:27:37.150681 2887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f657fdaacc946a5923777acbdc7f8d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43967 } }
I20260619 06:27:37.150924 2887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:37.151058 2887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:37.151207 2887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:37.153437 2887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f657fdaacc946a5923777acbdc7f8d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43967 } }
I20260619 06:27:37.153867 2887 leader_election.cc:302] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f657fdaacc946a5923777acbdc7f8d2; no voters:
I20260619 06:27:37.154371 2887 leader_election.cc:288] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:37.154491 2892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:37.155212 2892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [term 1 LEADER]: Becoming Leader. State: Replica: 9f657fdaacc946a5923777acbdc7f8d2, State: Running, Role: LEADER
I20260619 06:27:37.155759 2892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f657fdaacc946a5923777acbdc7f8d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43967 } }
I20260619 06:27:37.156219 2887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:37.162240 2893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f657fdaacc946a5923777acbdc7f8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f657fdaacc946a5923777acbdc7f8d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43967 } } }
I20260619 06:27:37.162314 2894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f657fdaacc946a5923777acbdc7f8d2. Latest consensus state: current_term: 1 leader_uuid: "9f657fdaacc946a5923777acbdc7f8d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f657fdaacc946a5923777acbdc7f8d2" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43967 } } }
I20260619 06:27:37.162667 2893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:37.162667 2894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2 [sys.catalog]: This master's current role is: LEADER
W20260619 06:27:37.166568 2906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:27:37.166684 2906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:27:37.166836 2907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:37.168524 2907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:37.176427 2907 catalog_manager.cc:1347] Generated new cluster ID: d883c3072ac7411296a04cf61d33db5c
I20260619 06:27:37.176522 2907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:37.190847 2907 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:37.192518 2907 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:37.207971 2907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f657fdaacc946a5923777acbdc7f8d2: Generated new TSK 0
I20260619 06:27:37.208933 2907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:37.255983 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43967
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:37.567611 2913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:37.567904 2913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:37.568058 2913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:37.575353 2913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:37.575500 2913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:37.575662 2913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:37.585552 2913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:37.586771 2913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:37.588027 2913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:37.600534 2918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:37.600569 2919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:37.600622 2921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:37.601697 2913 server_base.cc:1034] running on GCE node
I20260619 06:27:37.602331 2913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:37.603235 2913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:37.604457 2913 hybrid_clock.cc:648] HybridClock initialized: now 1781850457604426 us; error 45 us; skew 500 ppm
I20260619 06:27:37.604756 2913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:37 dist-test-slave-4t1f krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:37.607868 2913 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:37.609444 2913 webserver.cc:460] Webserver started at http://127.0.97.65:46293/ using document root <none> and password file <none>
I20260619 06:27:37.610111 2913 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:37.610253 2913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:37.610548 2913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:37.613382 2913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "4a42c73e32614948b2bfa1ff6adc5a5d"
format_stamp: "Formatted at 2026-06-19 06:27:37 on dist-test-slave-4t1f"
server_key: "d69e725c176ad1347ae3c570f0fa537c"
server_key_iv: "8df5ef21219beaccf3bce32db7c0fa35"
server_key_version: "encryptionkey@0"
I20260619 06:27:37.614125 2913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "4a42c73e32614948b2bfa1ff6adc5a5d"
format_stamp: "Formatted at 2026-06-19 06:27:37 on dist-test-slave-4t1f"
server_key: "d69e725c176ad1347ae3c570f0fa537c"
server_key_iv: "8df5ef21219beaccf3bce32db7c0fa35"
server_key_version: "encryptionkey@0"
I20260619 06:27:37.618657 2913 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:37.621999 2928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:37.623145 2913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:37.623314 2913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "4a42c73e32614948b2bfa1ff6adc5a5d"
format_stamp: "Formatted at 2026-06-19 06:27:37 on dist-test-slave-4t1f"
server_key: "d69e725c176ad1347ae3c570f0fa537c"
server_key_iv: "8df5ef21219beaccf3bce32db7c0fa35"
server_key_version: "encryptionkey@0"
I20260619 06:27:37.623500 2913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:37.648931 2913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:37.652804 2913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:37.653196 2913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:37.654044 2913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:37.655216 2913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:37.655314 2913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:37.655457 2913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:37.655529 2913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:37.688357 2913 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:33087
I20260619 06:27:37.688437 3040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:33087 every 8 connection(s)
I20260619 06:27:37.689672 2913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:37.693024 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 2913
I20260619 06:27:37.693245 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:37.694079 389 external_mini_cluster.cc:1428] Setting key fcb458763d40fb1e50c9ef5adad07956
Jun 19 06:27:37 dist-test-slave-4t1f krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:37.712538 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:37.691901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:60055 (local address 127.0.97.126:43967)
0619 06:27:37.692295 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:37.692299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:37.693317 (+ 1018us) server_negotiation.cc:407] Connection header received
0619 06:27:37.695661 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:37.695666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:37.695747 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:37.695906 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:37.697881 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:37.698590 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:37.701098 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:37.701631 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:37.701764 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:37.705613 (+ 3849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:37.705622 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:37.705630 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:37.705671 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:37.709384 (+ 3713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:37.710035 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:37.710044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:37.710050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:37.710220 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:37.710686 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:37.710693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:37.710699 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:37.711062 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:37.711213 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:37.711892 (+ 679us) server_negotiation.cc:299] Negotiation successful
0619 06:27:37.712120 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":218,"threads_started":1}
I20260619 06:27:37.713263 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:37.692303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43967 (local address 127.0.97.65:60055)
0619 06:27:37.693142 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:37.693197 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:37.695421 (+ 2224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:37.696152 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:37.696164 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:37.696879 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:37.697706 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:37.697733 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:37.698742 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:37.698750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:37.700926 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:37.700944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:37.701849 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:37.701858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:37.702101 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:37.702767 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:37.702799 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:37.705372 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:37.709507 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:37.709513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:37.709524 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:37.709897 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:37.710346 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:37.710354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:37.710360 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:37.710566 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:37.711182 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:37.711187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:37.711497 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 06:27:37.711720 (+ 223us) client_negotiation.cc:240] Negotiation successful
0619 06:27:37.712071 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":300,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260619 06:27:37.714727 3041 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43967
I20260619 06:27:37.715009 3041 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:37.715744 3041 heartbeater.cc:511] Master 127.0.97.126:43967 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260619 06:27:37.717954 2852 ts_manager.cc:194] Registered new tserver with Master: 4a42c73e32614948b2bfa1ff6adc5a5d (127.0.97.65:33087)
I20260619 06:27:37.719740 2852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:60055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:27:37.735836 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43967
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:38.038365 3047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:38.038681 3047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:38.038838 3047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:38.045812 3047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:38.045956 3047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:38.046123 3047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:38.055397 3047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:38.056663 3047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:38.057945 3047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:38.070858 3052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:38.070945 3055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:38.070879 3053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:38.072057 3047 server_base.cc:1034] running on GCE node
I20260619 06:27:38.072767 3047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:38.074020 3047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:38.075309 3047 hybrid_clock.cc:648] HybridClock initialized: now 1781850458075228 us; error 67 us; skew 500 ppm
I20260619 06:27:38.075730 3047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:38 dist-test-slave-4t1f krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:38.079991 3047 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:38.082041 3047 webserver.cc:460] Webserver started at http://127.0.97.66:44725/ using document root <none> and password file <none>
I20260619 06:27:38.082739 3047 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:38.082880 3047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:38.083204 3047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:38.086131 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "72b8d291452543998ce04caea21750af"
format_stamp: "Formatted at 2026-06-19 06:27:38 on dist-test-slave-4t1f"
server_key: "3db6f49896e1b27441b9363f7a0855c4"
server_key_iv: "cc930722b1063febb38a34947b5dd408"
server_key_version: "encryptionkey@0"
I20260619 06:27:38.086886 3047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "72b8d291452543998ce04caea21750af"
format_stamp: "Formatted at 2026-06-19 06:27:38 on dist-test-slave-4t1f"
server_key: "3db6f49896e1b27441b9363f7a0855c4"
server_key_iv: "cc930722b1063febb38a34947b5dd408"
server_key_version: "encryptionkey@0"
I20260619 06:27:38.091789 3047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:27:38.095060 3062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:38.096328 3047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:27:38.096503 3047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "72b8d291452543998ce04caea21750af"
format_stamp: "Formatted at 2026-06-19 06:27:38 on dist-test-slave-4t1f"
server_key: "3db6f49896e1b27441b9363f7a0855c4"
server_key_iv: "cc930722b1063febb38a34947b5dd408"
server_key_version: "encryptionkey@0"
I20260619 06:27:38.096698 3047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:38.120126 3047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:38.124275 3047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:38.124544 3047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:38.125517 3047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:38.126884 3047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:38.126998 3047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:38.127151 3047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:38.127223 3047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:38.159052 3047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41775
I20260619 06:27:38.159099 3174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41775 every 8 connection(s)
I20260619 06:27:38.160346 3047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:38.170682 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3047
I20260619 06:27:38.170852 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:38.171311 389 external_mini_cluster.cc:1428] Setting key 179cdeb2bccb985e6b931c1550227fee
Jun 19 06:27:38 dist-test-slave-4t1f krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:38.179582 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:38.162756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51997 (local address 127.0.97.126:43967)
0619 06:27:38.162920 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:38.162925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:38.164094 (+ 1169us) server_negotiation.cc:407] Connection header received
0619 06:27:38.165942 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:38.165948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:38.166036 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:38.166152 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:38.167585 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.168306 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.169858 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.170294 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.170425 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:38.174159 (+ 3734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:38.174164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:38.174167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:38.174196 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:38.176703 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:38.177480 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:38.177488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:38.177496 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:38.177605 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:38.178013 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:38.178018 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:38.178021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:38.178273 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:38.178421 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:38.179054 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 06:27:38.179278 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:27:38.180503 3177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:38.163263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43967 (local address 127.0.97.66:51997)
0619 06:27:38.163938 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:38.163973 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:38.165683 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:38.166380 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:38.166388 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:38.166887 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:38.167426 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:38.167442 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.168447 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.168454 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:38.169719 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:38.169730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.170552 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.170560 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:38.170879 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:38.171621 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:38.171642 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:38.173949 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:38.176837 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:38.176846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:38.176861 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:38.177360 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:38.177713 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:38.177718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:38.177722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:38.177898 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:38.178400 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:38.178407 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:38.178715 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 06:27:38.178963 (+ 248us) client_negotiation.cc:240] Negotiation successful
0619 06:27:38.179455 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":277,"threads_started":1}
I20260619 06:27:38.181965 3175 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43967
I20260619 06:27:38.182265 3175 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:38.182902 3175 heartbeater.cc:511] Master 127.0.97.126:43967 requested a full tablet report, sending...
I20260619 06:27:38.184243 2852 ts_manager.cc:194] Registered new tserver with Master: 72b8d291452543998ce04caea21750af (127.0.97.66:41775)
I20260619 06:27:38.185001 2852 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:51997
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:38.211670 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43967
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:38.532248 3180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:38.532490 3180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:38.532645 3180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:38.539505 3180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:38.539613 3180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:38.539765 3180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:38.548815 3180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43967
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:38.549994 3180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:38.551244 3180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:38.564455 3185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:38.564486 3188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:38.565080 3186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:38.565229 3180 server_base.cc:1034] running on GCE node
I20260619 06:27:38.565886 3180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:38.566661 3180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:38.567852 3180 hybrid_clock.cc:648] HybridClock initialized: now 1781850458567829 us; error 36 us; skew 500 ppm
I20260619 06:27:38.568154 3180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:38 dist-test-slave-4t1f krb5kdc[2807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:38.571156 3180 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:38.572576 3180 webserver.cc:460] Webserver started at http://127.0.97.67:36957/ using document root <none> and password file <none>
I20260619 06:27:38.573271 3180 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:38.573414 3180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:38.573810 3180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:38.576524 3180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "f6a3fa5faf82480ca0623bd78cce675e"
format_stamp: "Formatted at 2026-06-19 06:27:38 on dist-test-slave-4t1f"
server_key: "c24806a21b61af8a44314d92159f0dc5"
server_key_iv: "79554cb1dd340c64738b5fa634d4d5e8"
server_key_version: "encryptionkey@0"
I20260619 06:27:38.577335 3180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "f6a3fa5faf82480ca0623bd78cce675e"
format_stamp: "Formatted at 2026-06-19 06:27:38 on dist-test-slave-4t1f"
server_key: "c24806a21b61af8a44314d92159f0dc5"
server_key_iv: "79554cb1dd340c64738b5fa634d4d5e8"
server_key_version: "encryptionkey@0"
I20260619 06:27:38.581998 3180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:27:38.585230 3195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:38.586406 3180 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:27:38.586556 3180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "f6a3fa5faf82480ca0623bd78cce675e"
format_stamp: "Formatted at 2026-06-19 06:27:38 on dist-test-slave-4t1f"
server_key: "c24806a21b61af8a44314d92159f0dc5"
server_key_iv: "79554cb1dd340c64738b5fa634d4d5e8"
server_key_version: "encryptionkey@0"
I20260619 06:27:38.586747 3180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:38.612357 3180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:38.615952 3180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:38.616217 3180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:38.617167 3180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:38.618387 3180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:38.618489 3180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:38.618635 3180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:38.618707 3180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:38.649529 3180 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44465
I20260619 06:27:38.649569 3307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44465 every 8 connection(s)
I20260619 06:27:38.650705 3180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:38.657028 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3180
I20260619 06:27:38.657213 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:38.657680 389 external_mini_cluster.cc:1428] Setting key e8622c88314b85a06e1b67b83fb527ef
Jun 19 06:27:38 dist-test-slave-4t1f krb5kdc[2807](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:38.669659 3043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:38.652969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55395 (local address 127.0.97.126:43967)
0619 06:27:38.653159 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:38.653182 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:38.654231 (+ 1049us) server_negotiation.cc:407] Connection header received
0619 06:27:38.656033 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:38.656038 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:38.656122 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:38.656240 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:38.658117 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.659115 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.661161 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.661545 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.661634 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:38.664804 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:38.664809 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:38.664812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:38.664842 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:38.667266 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:38.667870 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:38.667874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:38.667876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:38.667976 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:38.668325 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:38.668328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:38.668330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:38.668562 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:38.668662 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:38.669287 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 06:27:38.669440 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:27:38.670554 3310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:38.653429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43967 (local address 127.0.97.67:55395)
0619 06:27:38.654067 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:38.654108 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:38.655742 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:38.656451 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:38.656459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:38.657214 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:38.657953 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:38.657977 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.659371 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.659377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:38.660993 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:38.661009 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:38.661716 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:38.661721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:38.661892 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:38.662540 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:38.662561 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:38.664615 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:38.667377 (+ 2762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:38.667383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:38.667393 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:38.667770 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:38.668060 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:38.668063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:38.668065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:38.668231 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:38.668656 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:38.668661 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:38.668934 (+ 273us) client_negotiation.cc:769] Sending connection context
0619 06:27:38.669213 (+ 279us) client_negotiation.cc:240] Negotiation successful
0619 06:27:38.669545 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":261,"threads_started":1}
I20260619 06:27:38.671947 3308 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43967
I20260619 06:27:38.672272 3308 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:38.672942 3308 heartbeater.cc:511] Master 127.0.97.126:43967 requested a full tablet report, sending...
I20260619 06:27:38.674292 2851 ts_manager.cc:194] Registered new tserver with Master: f6a3fa5faf82480ca0623bd78cce675e (127.0.97.67:44465)
I20260619 06:27:38.675062 2851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:55395
I20260619 06:27:38.684508 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:27:38.712457 3043 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:27:38.710143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45864 (local address 127.0.97.126:43967)
0619 06:27:38.710389 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:38.710414 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:38.710687 (+ 273us) server_negotiation.cc:407] Connection header received
0619 06:27:38.710832 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:38.710837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:38.710899 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:38.710985 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:38.712168 (+ 1183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45864: BlockingRecv error: recv got EOF from 127.0.0.1:45864 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:27:38.715379 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2913
W20260619 06:27:38.717929 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2913: Not authorized: Client connection negotiation failed: client connection to 127.0.97.65:33087: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:27:38.718127 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 2913
I20260619 06:27:38.732961 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3047
W20260619 06:27:38.735195 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3047: Not authorized: Client connection negotiation failed: client connection to 127.0.97.66:41775: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:27:38.735320 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3047
I20260619 06:27:38.750011 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3180
W20260619 06:27:38.752581 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3180: Not authorized: Client connection negotiation failed: client connection to 127.0.97.67:44465: server requires authentication, but client does not have Kerberos credentials available
I20260619 06:27:38.752720 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3180
I20260619 06:27:38.767601 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 2820
I20260619 06:27:38.768577 2882 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:38.869114 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 2820
2026-06-19T06:27:38Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3408 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:38 dist-test-slave-4t1f krb5kdc[3325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:38 dist-test-slave-4t1f krb5kdc[3325](info): set up 2 sockets
Jun 19 06:27:38 dist-test-slave-4t1f krb5kdc[3325](info): commencing operation
krb5kdc: starting...
W20260619 06:27:39.937304 389 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:40 dist-test-slave-4t1f krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:40.121980 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42015
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:40.427331 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:40.427616 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:40.427702 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:40.434631 3338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:40.434729 3338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:40.434772 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:40.434834 3338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:40.434880 3338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:40.444272 3338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:40.445472 3338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:40.446657 3338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:40.458630 3343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:40.458595 3346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:40.458670 3344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:40.459821 3338 server_base.cc:1034] running on GCE node
I20260619 06:27:40.460592 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:40.461971 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:40.463191 3338 hybrid_clock.cc:648] HybridClock initialized: now 1781850460463162 us; error 46 us; skew 500 ppm
I20260619 06:27:40.463522 3338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:40 dist-test-slave-4t1f krb5kdc[3325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:40.469594 3338 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:40.471025 3338 webserver.cc:460] Webserver started at http://127.0.97.126:35567/ using document root <none> and password file <none>
I20260619 06:27:40.471695 3338 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:40.471840 3338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:40.472132 3338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:40.475041 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "587631be8d664373b3f5599cccaeccb5"
format_stamp: "Formatted at 2026-06-19 06:27:40 on dist-test-slave-4t1f"
server_key: "81e8e11595e585cd16ce224281eb5d65"
server_key_iv: "910ec92f0028d1abaf451e7a773d7e0d"
server_key_version: "encryptionkey@0"
I20260619 06:27:40.475795 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "587631be8d664373b3f5599cccaeccb5"
format_stamp: "Formatted at 2026-06-19 06:27:40 on dist-test-slave-4t1f"
server_key: "81e8e11595e585cd16ce224281eb5d65"
server_key_iv: "910ec92f0028d1abaf451e7a773d7e0d"
server_key_version: "encryptionkey@0"
I20260619 06:27:40.480861 3338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260619 06:27:40.484287 3353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:40.485625 3338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:40.485802 3338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "587631be8d664373b3f5599cccaeccb5"
format_stamp: "Formatted at 2026-06-19 06:27:40 on dist-test-slave-4t1f"
server_key: "81e8e11595e585cd16ce224281eb5d65"
server_key_iv: "910ec92f0028d1abaf451e7a773d7e0d"
server_key_version: "encryptionkey@0"
I20260619 06:27:40.485998 3338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:40.510079 3338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:40.514258 3338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:40.514557 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:40.533808 3338 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42015
I20260619 06:27:40.533897 3404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42015 every 8 connection(s)
I20260619 06:27:40.535120 3338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:40.538041 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3338
I20260619 06:27:40.538261 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:40.539065 389 external_mini_cluster.cc:1428] Setting key abc2cb3fbfcfafe73ce40868abc1774f
I20260619 06:27:40.539252 3405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:40.551627 3405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5: Bootstrap starting.
Jun 19 06:27:40 dist-test-slave-4t1f krb5kdc[3325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850460, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:40.554949 3405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:40.556200 3405 log.cc:824] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:40.559383 3405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5: No bootstrap required, opened a new log
I20260619 06:27:40.560561 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:40.540922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42400 (local address 127.0.97.126:42015)
0619 06:27:40.541662 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:40.541673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:40.541713 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:27:40.543686 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:40.543709 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:40.544078 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:40.544456 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:40.545999 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:40.547256 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:40.548517 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:40.549289 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:40.549516 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:40.553322 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:40.553332 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:40.553348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:40.553380 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:40.556223 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:40.556919 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:40.556926 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:40.556932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:40.557161 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:40.557593 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:40.557598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:40.557601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:40.558255 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:40.558541 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:40.558864 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 06:27:40.559260 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":349,"threads_started":1}
I20260619 06:27:40.564288 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:27:40.567744 3405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587631be8d664373b3f5599cccaeccb5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42015 } }
I20260619 06:27:40.568188 3405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:40.568346 3405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 587631be8d664373b3f5599cccaeccb5, State: Initialized, Role: FOLLOWER
I20260619 06:27:40.569021 3405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587631be8d664373b3f5599cccaeccb5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42015 } }
I20260619 06:27:40.569346 3405 raft_consensus.cc:388] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:40.569511 3405 raft_consensus.cc:482] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:40.569689 3405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:40.603549 3408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:40.597268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.0.97.126:42015)
0619 06:27:40.597869 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:40.597873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:40.597892 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:27:40.597949 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:40.597952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:40.598012 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:40.598097 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:40.598999 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:40.599654 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:40.600821 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:40.601264 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:40.601354 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:40.602242 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:40.602246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:40.602249 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:40.602882 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:40.602935 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:40.602956 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:40.603040 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:40.603144 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 06:27:40.603212 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478}
I20260619 06:27:40.611642 3405 raft_consensus.cc:504] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587631be8d664373b3f5599cccaeccb5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42015 } }
I20260619 06:27:40.612334 3405 leader_election.cc:302] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 587631be8d664373b3f5599cccaeccb5; no voters:
I20260619 06:27:40.613021 3405 leader_election.cc:288] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:40.613184 3417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:40.613833 3417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [term 1 LEADER]: Becoming Leader. State: Replica: 587631be8d664373b3f5599cccaeccb5, State: Running, Role: LEADER
I20260619 06:27:40.614526 3417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587631be8d664373b3f5599cccaeccb5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42015 } }
I20260619 06:27:40.616712 3405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:40.619052 3419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "587631be8d664373b3f5599cccaeccb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587631be8d664373b3f5599cccaeccb5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42015 } } }
I20260619 06:27:40.619082 3418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 587631be8d664373b3f5599cccaeccb5. Latest consensus state: current_term: 1 leader_uuid: "587631be8d664373b3f5599cccaeccb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "587631be8d664373b3f5599cccaeccb5" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42015 } } }
I20260619 06:27:40.619496 3419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:40.619589 3418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:40.620301 3426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:40.625809 3426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:40.633351 3426 catalog_manager.cc:1347] Generated new cluster ID: da3d655243d64b9791aa993a1d06778a
I20260619 06:27:40.633445 3426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:40.664511 3426 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:40.665853 3426 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:40.674108 3426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 587631be8d664373b3f5599cccaeccb5: Generated new TSK 0
I20260619 06:27:40.675115 3426 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:27:40.694223 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3338
I20260619 06:27:40.695878 3400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:40.797436 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3338
2026-06-19T06:27:40Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1923 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:40 dist-test-slave-4t1f krb5kdc[3439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:40 dist-test-slave-4t1f krb5kdc[3439](info): set up 2 sockets
Jun 19 06:27:40 dist-test-slave-4t1f krb5kdc[3439](info): commencing operation
krb5kdc: starting...
W20260619 06:27:41.917441 389 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:42 dist-test-slave-4t1f krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:42.098572 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42139
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42139
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:42.399359 3452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:42.399694 3452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:42.399854 3452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:42.406771 3452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:42.406904 3452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:42.407009 3452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:42.407105 3452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:42.407176 3452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:42.416467 3452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:42.417819 3452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:42.419037 3452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:42.430225 3458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:42.430316 3457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:42.430284 3460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:42.431005 3452 server_base.cc:1034] running on GCE node
I20260619 06:27:42.431696 3452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:42.433099 3452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:42.434326 3452 hybrid_clock.cc:648] HybridClock initialized: now 1781850462434244 us; error 53 us; skew 500 ppm
I20260619 06:27:42.434595 3452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:42 dist-test-slave-4t1f krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:42.437520 3452 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:42.439006 3452 webserver.cc:460] Webserver started at http://127.0.97.126:45053/ using document root <none> and password file <none>
I20260619 06:27:42.439697 3452 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:42.439831 3452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:42.440135 3452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:42.443070 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "108d26ced4044e2988e5b02074d49102"
format_stamp: "Formatted at 2026-06-19 06:27:42 on dist-test-slave-4t1f"
server_key: "83e46763ebc708e00fb154772340bf52"
server_key_iv: "b8dbbaea027fa558760f7386e17574b6"
server_key_version: "encryptionkey@0"
I20260619 06:27:42.443818 3452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "108d26ced4044e2988e5b02074d49102"
format_stamp: "Formatted at 2026-06-19 06:27:42 on dist-test-slave-4t1f"
server_key: "83e46763ebc708e00fb154772340bf52"
server_key_iv: "b8dbbaea027fa558760f7386e17574b6"
server_key_version: "encryptionkey@0"
I20260619 06:27:42.448666 3452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 06:27:42.451781 3467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:42.452992 3452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:42.453181 3452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "108d26ced4044e2988e5b02074d49102"
format_stamp: "Formatted at 2026-06-19 06:27:42 on dist-test-slave-4t1f"
server_key: "83e46763ebc708e00fb154772340bf52"
server_key_iv: "b8dbbaea027fa558760f7386e17574b6"
server_key_version: "encryptionkey@0"
I20260619 06:27:42.453373 3452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:42.475703 3452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:42.479929 3452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:42.480244 3452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:42.499289 3452 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42139
I20260619 06:27:42.499301 3518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42139 every 8 connection(s)
I20260619 06:27:42.500679 3452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:42.503921 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3452
I20260619 06:27:42.504204 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:42.504295 3519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:42.504891 389 external_mini_cluster.cc:1428] Setting key a9ce4d49c1ed22ca259b7e5d096a9578
I20260619 06:27:42.517406 3519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102: Bootstrap starting.
Jun 19 06:27:42 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:42.521013 3519 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:42.522167 3519 log.cc:824] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:42.524785 3519 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102: No bootstrap required, opened a new log
I20260619 06:27:42.528627 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:42.506972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.0.97.126:42139)
0619 06:27:42.507819 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:42.507831 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:42.507874 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:27:42.510054 (+ 2180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:42.510076 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:42.510494 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:42.510841 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:42.512490 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:42.513901 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:42.515645 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:42.516360 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:42.516501 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:42.521228 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:42.521236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:42.521250 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:42.521288 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:42.524314 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:42.525013 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:42.525021 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:42.525029 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:42.525256 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:42.525624 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:42.525628 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:42.525631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:42.526264 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:42.526563 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:42.526896 (+ 333us) server_negotiation.cc:299] Negotiation successful
0619 06:27:42.527327 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":417,"threads_started":1}
I20260619 06:27:42.533715 3519 raft_consensus.cc:348] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d26ced4044e2988e5b02074d49102" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42139 } }
I20260619 06:27:42.533999 3519 raft_consensus.cc:374] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:42.534067 3519 raft_consensus.cc:729] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 108d26ced4044e2988e5b02074d49102, State: Initialized, Role: FOLLOWER
I20260619 06:27:42.534597 3519 consensus_queue.cc:260] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d26ced4044e2988e5b02074d49102" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42139 } }
I20260619 06:27:42.534794 3519 raft_consensus.cc:388] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:42.534912 3519 raft_consensus.cc:482] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:42.535084 3519 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:42.537277 3519 raft_consensus.cc:504] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d26ced4044e2988e5b02074d49102" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42139 } }
I20260619 06:27:42.537659 3519 leader_election.cc:302] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 108d26ced4044e2988e5b02074d49102; no voters:
I20260619 06:27:42.538127 3519 leader_election.cc:288] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:42.538216 3524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:42.539165 3524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [term 1 LEADER]: Becoming Leader. State: Replica: 108d26ced4044e2988e5b02074d49102, State: Running, Role: LEADER
I20260619 06:27:42.539628 3524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d26ced4044e2988e5b02074d49102" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42139 } }
I20260619 06:27:42.539973 3519 sys_catalog.cc:564] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:42.542263 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 108d26ced4044e2988e5b02074d49102. Latest consensus state: current_term: 1 leader_uuid: "108d26ced4044e2988e5b02074d49102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d26ced4044e2988e5b02074d49102" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42139 } } }
I20260619 06:27:42.542594 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:42.542358 3525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "108d26ced4044e2988e5b02074d49102" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "108d26ced4044e2988e5b02074d49102" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42139 } } }
I20260619 06:27:42.542763 3525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:42.549566 3531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:42.554322 3531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:42.562088 3531 catalog_manager.cc:1347] Generated new cluster ID: af201493043f44d7bd166330a9064e75
I20260619 06:27:42.562207 3531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:42.585242 3531 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:42.586617 3531 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:42.592689 3531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 108d26ced4044e2988e5b02074d49102: Generated new TSK 0
I20260619 06:27:42.593542 3531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:42.640123 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42139
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:42.946066 3545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:42.946310 3545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:42.946401 3545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:42.953553 3545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:42.953653 3545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:42.953760 3545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:42.963181 3545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:42.964300 3545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:42.965570 3545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:42.977712 3550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:42.977770 3551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:42.977684 3553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:42.978847 3545 server_base.cc:1034] running on GCE node
I20260619 06:27:42.979374 3545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:42.980175 3545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:42.981361 3545 hybrid_clock.cc:648] HybridClock initialized: now 1781850462981313 us; error 46 us; skew 500 ppm
I20260619 06:27:42.981628 3545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:42 dist-test-slave-4t1f krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:42.984632 3545 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:42.985971 3545 webserver.cc:460] Webserver started at http://127.0.97.65:35103/ using document root <none> and password file <none>
I20260619 06:27:42.986614 3545 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:42.986738 3545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:42.987025 3545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:42.989797 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "2b29b48d169a4386be1f1b2fed097edc"
format_stamp: "Formatted at 2026-06-19 06:27:42 on dist-test-slave-4t1f"
server_key: "c5f3b2cdc64140e976741b3ecc49d2b6"
server_key_iv: "ed7893bbd633e2c840c00d02c115d893"
server_key_version: "encryptionkey@0"
I20260619 06:27:42.990566 3545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "2b29b48d169a4386be1f1b2fed097edc"
format_stamp: "Formatted at 2026-06-19 06:27:42 on dist-test-slave-4t1f"
server_key: "c5f3b2cdc64140e976741b3ecc49d2b6"
server_key_iv: "ed7893bbd633e2c840c00d02c115d893"
server_key_version: "encryptionkey@0"
I20260619 06:27:42.995355 3545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:27:42.998642 3560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:42.999986 3545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:27:43.000149 3545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "2b29b48d169a4386be1f1b2fed097edc"
format_stamp: "Formatted at 2026-06-19 06:27:42 on dist-test-slave-4t1f"
server_key: "c5f3b2cdc64140e976741b3ecc49d2b6"
server_key_iv: "ed7893bbd633e2c840c00d02c115d893"
server_key_version: "encryptionkey@0"
I20260619 06:27:43.000339 3545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:43.025830 3545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:43.030223 3545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:43.030534 3545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:43.031427 3545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:43.032714 3545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:43.032783 3545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:43.032943 3545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:43.033002 3545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:43.064945 3545 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36679
I20260619 06:27:43.064998 3672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36679 every 8 connection(s)
I20260619 06:27:43.066258 3545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:43.076258 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3545
I20260619 06:27:43.076507 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:43.077224 389 external_mini_cluster.cc:1428] Setting key efd998e7ec6b6ac35c5e3114e663f89c
Jun 19 06:27:43 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:43.086467 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:43.068786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39961 (local address 127.0.97.126:42139)
0619 06:27:43.069231 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:43.069235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:43.070081 (+ 846us) server_negotiation.cc:407] Connection header received
0619 06:27:43.071836 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:43.071841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:43.071927 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:43.072039 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:43.073501 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.074280 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.075802 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.076204 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.076338 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:43.080403 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:43.080408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:43.080411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:43.080439 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:43.082935 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:43.083897 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:43.083901 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:43.083903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:43.084009 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:43.084512 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:43.084518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:43.084523 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:43.084892 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:43.085045 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:43.085745 (+ 700us) server_negotiation.cc:299] Negotiation successful
0619 06:27:43.085984 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":225,"threads_started":1}
I20260619 06:27:43.087060 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:43.069252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42139 (local address 127.0.97.65:39961)
0619 06:27:43.069925 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:43.069968 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:43.071570 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:43.072265 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:43.072273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:43.072749 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:43.073343 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:43.073362 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.074426 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.074438 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:43.075649 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:43.075660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.076420 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.076429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:43.076721 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:43.077461 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:43.077483 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:43.080205 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:43.083075 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:43.083086 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:43.083107 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:43.083771 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:43.084118 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:43.084126 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:43.084132 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:43.084378 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:43.085034 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:43.085040 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:43.085378 (+ 338us) client_negotiation.cc:769] Sending connection context
0619 06:27:43.085604 (+ 226us) client_negotiation.cc:240] Negotiation successful
0619 06:27:43.085969 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":281,"threads_started":1}
I20260619 06:27:43.088542 3673 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42139
I20260619 06:27:43.088850 3673 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:43.089696 3673 heartbeater.cc:511] Master 127.0.97.126:42139 requested a full tablet report, sending...
I20260619 06:27:43.091974 3484 ts_manager.cc:194] Registered new tserver with Master: 2b29b48d169a4386be1f1b2fed097edc (127.0.97.65:36679)
I20260619 06:27:43.093572 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:39961
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:27:43.119390 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42139
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:43.420449 3679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:43.420687 3679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:43.420774 3679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:43.427526 3679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:43.427619 3679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:43.427729 3679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:43.436560 3679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:43.437671 3679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:43.438831 3679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:43.451346 3684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:43.451308 3685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:43.451392 3687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:43.454375 3679 server_base.cc:1034] running on GCE node
I20260619 06:27:43.454981 3679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:43.455821 3679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:43.457016 3679 hybrid_clock.cc:648] HybridClock initialized: now 1781850463456981 us; error 25 us; skew 500 ppm
I20260619 06:27:43.457530 3679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:43 dist-test-slave-4t1f krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:43.460379 3679 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:43.461779 3679 webserver.cc:460] Webserver started at http://127.0.97.66:34039/ using document root <none> and password file <none>
I20260619 06:27:43.462433 3679 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:43.462556 3679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:43.462904 3679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:43.465662 3679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "197e95aa7ce84eaabffc4eda04f93770"
format_stamp: "Formatted at 2026-06-19 06:27:43 on dist-test-slave-4t1f"
server_key: "6452ecbad5865c0d0173ebcf7c8e2838"
server_key_iv: "402c121e3c751f7cc2fbc3b2bc735727"
server_key_version: "encryptionkey@0"
I20260619 06:27:43.466396 3679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "197e95aa7ce84eaabffc4eda04f93770"
format_stamp: "Formatted at 2026-06-19 06:27:43 on dist-test-slave-4t1f"
server_key: "6452ecbad5865c0d0173ebcf7c8e2838"
server_key_iv: "402c121e3c751f7cc2fbc3b2bc735727"
server_key_version: "encryptionkey@0"
I20260619 06:27:43.471131 3679 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:43.474321 3694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:43.475509 3679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 06:27:43.475675 3679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "197e95aa7ce84eaabffc4eda04f93770"
format_stamp: "Formatted at 2026-06-19 06:27:43 on dist-test-slave-4t1f"
server_key: "6452ecbad5865c0d0173ebcf7c8e2838"
server_key_iv: "402c121e3c751f7cc2fbc3b2bc735727"
server_key_version: "encryptionkey@0"
I20260619 06:27:43.475869 3679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:43.495054 3679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:43.498900 3679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:43.499171 3679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:43.500062 3679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:43.501361 3679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:43.501456 3679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:43.501617 3679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:43.501688 3679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:43.534413 3679 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38655
I20260619 06:27:43.534462 3806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38655 every 8 connection(s)
I20260619 06:27:43.535637 3679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:43.544160 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3679
I20260619 06:27:43.544368 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:43.544767 389 external_mini_cluster.cc:1428] Setting key 4e78c690ffac76272b59c1e556a40212
Jun 19 06:27:43 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:43.554857 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:43.538159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52343 (local address 127.0.97.126:42139)
0619 06:27:43.538326 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:43.538330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:43.539458 (+ 1128us) server_negotiation.cc:407] Connection header received
0619 06:27:43.541315 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:43.541320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:43.541406 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:43.541509 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:43.542917 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.543638 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.545421 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.545943 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.546042 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:43.549596 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:43.549601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:43.549604 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:43.549636 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:43.552059 (+ 2423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:43.552774 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:43.552779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:43.552782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:43.552901 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:43.553356 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:43.553360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:43.553364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:43.553589 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:43.553729 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:43.554324 (+ 595us) server_negotiation.cc:299] Negotiation successful
0619 06:27:43.554522 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":63}
I20260619 06:27:43.555773 3809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:43.538660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42139 (local address 127.0.97.66:52343)
0619 06:27:43.539310 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:43.539353 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:43.540995 (+ 1642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:43.541731 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:43.541740 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:43.542223 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:43.542754 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:43.542770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.543788 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.543795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:43.545261 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:43.545276 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:43.546163 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:43.546169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:43.546471 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:43.547238 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:43.547262 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:43.549395 (+ 2133us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:43.552232 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:43.552240 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:43.552254 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:43.552632 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:43.553007 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:43.553013 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:43.553016 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:43.553244 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:43.553710 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:43.553718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:43.554008 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 06:27:43.554241 (+ 233us) client_negotiation.cc:240] Negotiation successful
0619 06:27:43.554710 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":257,"threads_started":1}
I20260619 06:27:43.557185 3807 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42139
I20260619 06:27:43.557463 3807 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:43.558228 3807 heartbeater.cc:511] Master 127.0.97.126:42139 requested a full tablet report, sending...
I20260619 06:27:43.559648 3484 ts_manager.cc:194] Registered new tserver with Master: 197e95aa7ce84eaabffc4eda04f93770 (127.0.97.66:38655)
I20260619 06:27:43.560825 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:52343
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:43.584401 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42139
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:43.889110 3812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:43.889356 3812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:43.889505 3812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:43.896312 3812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:43.896433 3812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:43.896634 3812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:43.906023 3812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42139
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:43.907161 3812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:43.908376 3812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:43.920881 3817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:43.920925 3818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:43.920953 3820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:43.921878 3812 server_base.cc:1034] running on GCE node
I20260619 06:27:43.922458 3812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:43.923305 3812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:43.924506 3812 hybrid_clock.cc:648] HybridClock initialized: now 1781850463924468 us; error 35 us; skew 500 ppm
I20260619 06:27:43.924858 3812 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:43 dist-test-slave-4t1f krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:43.928004 3812 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:43.929395 3812 webserver.cc:460] Webserver started at http://127.0.97.67:46721/ using document root <none> and password file <none>
I20260619 06:27:43.930028 3812 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:43.930154 3812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:43.930459 3812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:43.933372 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "d184bb4017bf45c3a9b2a7e3697077aa"
format_stamp: "Formatted at 2026-06-19 06:27:43 on dist-test-slave-4t1f"
server_key: "aff566f1f015404004884ad83b33fa71"
server_key_iv: "899cc2a328198a07dc69e6c3952a958d"
server_key_version: "encryptionkey@0"
I20260619 06:27:43.934119 3812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "d184bb4017bf45c3a9b2a7e3697077aa"
format_stamp: "Formatted at 2026-06-19 06:27:43 on dist-test-slave-4t1f"
server_key: "aff566f1f015404004884ad83b33fa71"
server_key_iv: "899cc2a328198a07dc69e6c3952a958d"
server_key_version: "encryptionkey@0"
I20260619 06:27:43.938870 3812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 06:27:43.942166 3827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:43.943437 3812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:27:43.943596 3812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "d184bb4017bf45c3a9b2a7e3697077aa"
format_stamp: "Formatted at 2026-06-19 06:27:43 on dist-test-slave-4t1f"
server_key: "aff566f1f015404004884ad83b33fa71"
server_key_iv: "899cc2a328198a07dc69e6c3952a958d"
server_key_version: "encryptionkey@0"
I20260619 06:27:43.943783 3812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:43.964090 3812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:43.968077 3812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:43.968338 3812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:43.969358 3812 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:43.970584 3812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:43.970655 3812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:43.970729 3812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:43.970780 3812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:44.002846 3812 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39881
I20260619 06:27:44.002930 3939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39881 every 8 connection(s)
I20260619 06:27:44.004103 3812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:44.010301 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3812
I20260619 06:27:44.010515 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:44.010938 389 external_mini_cluster.cc:1428] Setting key 85df4cdbda3f6a6a2ea260f21119d05b
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:44.022329 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:44.006612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42683 (local address 127.0.97.126:42139)
0619 06:27:44.006791 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:44.006795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:44.007811 (+ 1016us) server_negotiation.cc:407] Connection header received
0619 06:27:44.009566 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:44.009571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:44.009644 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:44.009778 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:44.011273 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.012286 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.013788 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.014239 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.014332 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:44.017521 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:44.017526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:44.017529 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:44.017559 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:44.019896 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.020516 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.020520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.020522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.020631 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.020976 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.020979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.020981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.021282 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:44.021405 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:44.021966 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 06:27:44.022107 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:27:44.023270 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:44.006994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42139 (local address 127.0.97.67:42683)
0619 06:27:44.007654 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:44.007689 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:44.009291 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:44.009957 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:44.009965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:44.010420 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:44.011089 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:44.011116 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.012414 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.012418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:44.013670 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:44.013681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.014409 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.014414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:44.014609 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:44.015259 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:44.015290 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:44.017354 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:44.020019 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:44.020025 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:44.020036 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:44.020414 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:44.020717 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:44.020721 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:44.020723 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:44.020883 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:44.021389 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:44.021395 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:44.021669 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 06:27:44.021891 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 06:27:44.022231 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
I20260619 06:27:44.024634 3940 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42139
I20260619 06:27:44.024940 3940 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:44.025646 3940 heartbeater.cc:511] Master 127.0.97.126:42139 requested a full tablet report, sending...
I20260619 06:27:44.026922 3484 ts_manager.cc:194] Registered new tserver with Master: d184bb4017bf45c3a9b2a7e3697077aa (127.0.97.67:39881)
I20260619 06:27:44.027729 3484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:42683
I20260619 06:27:44.037719 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:44.086442 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:44.070477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40270 (local address 127.0.97.126:42139)
0619 06:27:44.070707 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:44.070716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:44.071081 (+ 365us) server_negotiation.cc:407] Connection header received
0619 06:27:44.071292 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:44.071298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:44.071358 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:44.071459 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:44.072749 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.073570 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.075730 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.076507 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.076679 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:44.081709 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:44.081714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:44.081717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:44.081749 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:44.084260 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.084788 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.084815 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.084817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.084930 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.085287 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.085291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.085294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.085587 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:44.085746 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:44.085958 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:27:44.086142 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 06:27:44.086993 3484 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40270
I20260619 06:27:44.088507 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3545
I20260619 06:27:44.096886 3673 heartbeater.cc:503] Master 127.0.97.126:42139 was elected leader, sending a full tablet report...
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:44.103802 3951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:44.089645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.0.97.65:36679)
0619 06:27:44.090260 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:44.090267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:44.090289 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:27:44.090337 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:44.090343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:44.090540 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:44.090653 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:44.091836 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.093161 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.094307 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.094871 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.095008 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:44.098923 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:44.098932 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:44.098945 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:44.098977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:44.101689 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.102219 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.102225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.102227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.102340 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.102602 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.102604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.102606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.102902 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:44.103095 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:44.103332 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 06:27:44.103545 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":337,"threads_started":1}
W20260619 06:27:44.104941 3668 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37766
W20260619 06:27:44.106026 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3545: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:44.106181 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3545
I20260619 06:27:44.121771 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3679
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:27:44.136862 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:44.122492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.0.97.66:38655)
0619 06:27:44.123018 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:44.123025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:44.123046 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:27:44.123119 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:44.123125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:44.123324 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:44.123485 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:44.124628 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.125717 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.126960 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.127474 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.127608 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:44.131636 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:44.131645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:44.131657 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:44.131693 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:44.134400 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.135071 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.135077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.135079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.135197 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.135489 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.135493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.135495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.135782 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:44.135995 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:44.136305 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 06:27:44.136582 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":247,"threads_started":1}
W20260619 06:27:44.138021 3802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36690
W20260619 06:27:44.139021 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3679: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:44.139143 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3679
I20260619 06:27:44.153882 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3812
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:27:44.169790 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:44.154828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.0.97.67:39881)
0619 06:27:44.155428 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:44.155435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:44.155462 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:27:44.155517 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:44.155524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:44.155726 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:44.155856 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:44.157088 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.158248 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.159329 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:44.159830 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:44.159919 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:44.163945 (+ 4026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:44.163956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:44.163970 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:44.164003 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:44.166797 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.167545 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.167559 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.167563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.167685 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:44.168063 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:44.168070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:44.168073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:44.168430 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:44.168779 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:44.169143 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 06:27:44.169422 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":319,"threads_started":1}
W20260619 06:27:44.171092 3935 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40788
W20260619 06:27:44.172197 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3812: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:27:44.172389 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3812
I20260619 06:27:44.187278 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3452
I20260619 06:27:44.188339 3514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:44.288054 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3452
2026-06-19T06:27:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3495 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3957](info): set up 2 sockets
Jun 19 06:27:44 dist-test-slave-4t1f krb5kdc[3957](info): commencing operation
krb5kdc: starting...
W20260619 06:27:45.373593 389 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:45 dist-test-slave-4t1f krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:45.552907 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45719
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:45.854135 3970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:45.854394 3970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:45.854563 3970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:45.861683 3970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:45.861775 3970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:45.861889 3970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:45.861986 3970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:45.862066 3970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:45.871276 3970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:45.872421 3970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:45.873658 3970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:45.885337 3978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:45.885416 3976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:45.885813 3975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:45.886531 3970 server_base.cc:1034] running on GCE node
I20260619 06:27:45.887238 3970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:45.888465 3970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:45.889695 3970 hybrid_clock.cc:648] HybridClock initialized: now 1781850465889662 us; error 45 us; skew 500 ppm
I20260619 06:27:45.890027 3970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:45 dist-test-slave-4t1f krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:45.896065 3970 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:45.897531 3970 webserver.cc:460] Webserver started at http://127.0.97.126:35817/ using document root <none> and password file <none>
I20260619 06:27:45.898190 3970 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:45.898324 3970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:45.898633 3970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:45.901439 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "789974ee223246968578b682c29ff5ef"
format_stamp: "Formatted at 2026-06-19 06:27:45 on dist-test-slave-4t1f"
server_key: "1cfb34a613918fb8805c0b32a90ca45a"
server_key_iv: "60d0de3247a6657ebe3dc3a94fad9d70"
server_key_version: "encryptionkey@0"
I20260619 06:27:45.902179 3970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "789974ee223246968578b682c29ff5ef"
format_stamp: "Formatted at 2026-06-19 06:27:45 on dist-test-slave-4t1f"
server_key: "1cfb34a613918fb8805c0b32a90ca45a"
server_key_iv: "60d0de3247a6657ebe3dc3a94fad9d70"
server_key_version: "encryptionkey@0"
I20260619 06:27:45.906841 3970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:45.909934 3985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:45.911128 3970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:27:45.911298 3970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "789974ee223246968578b682c29ff5ef"
format_stamp: "Formatted at 2026-06-19 06:27:45 on dist-test-slave-4t1f"
server_key: "1cfb34a613918fb8805c0b32a90ca45a"
server_key_iv: "60d0de3247a6657ebe3dc3a94fad9d70"
server_key_version: "encryptionkey@0"
I20260619 06:27:45.911473 3970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:45.933203 3970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:45.937202 3970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:45.937471 3970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:45.954707 3970 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45719
I20260619 06:27:45.954782 4036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45719 every 8 connection(s)
I20260619 06:27:45.955893 3970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:45.957731 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 3970
I20260619 06:27:45.957944 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:45.958571 389 external_mini_cluster.cc:1428] Setting key 36d11e8c39bba592aa76211883268e70
I20260619 06:27:45.959398 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:27:45 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:45.972102 4037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef: Bootstrap starting.
I20260619 06:27:45.975853 4037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:45.977002 4037 log.cc:824] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:45.980044 4037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef: No bootstrap required, opened a new log
I20260619 06:27:45.980537 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:45.960369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.0.97.126:45719)
0619 06:27:45.961171 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:45.961185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:45.961233 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:27:45.962762 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:45.962786 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:45.963240 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:45.963665 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:45.964840 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:45.966863 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:45.967988 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:45.968978 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:45.969205 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:45.972756 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:45.972766 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:45.972781 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:45.972819 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:45.976562 (+ 3743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:45.977184 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:45.977193 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:45.977200 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:45.977387 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:45.977807 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:45.977810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:45.977812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:45.978310 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:45.978591 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:45.978929 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 06:27:45.979299 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":328,"threads_started":1}
I20260619 06:27:45.990350 4037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789974ee223246968578b682c29ff5ef" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45719 } }
I20260619 06:27:45.990784 4037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:45.990904 4037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789974ee223246968578b682c29ff5ef, State: Initialized, Role: FOLLOWER
I20260619 06:27:45.991531 4037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789974ee223246968578b682c29ff5ef" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45719 } }
I20260619 06:27:45.991818 4037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:45.992007 4037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:45.992187 4037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:46.017262 4037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789974ee223246968578b682c29ff5ef" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45719 } }
I20260619 06:27:46.017997 4037 leader_election.cc:302] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 789974ee223246968578b682c29ff5ef; no voters:
I20260619 06:27:46.018738 4037 leader_election.cc:288] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:46.018896 4042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:46.019764 4042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [term 1 LEADER]: Becoming Leader. State: Replica: 789974ee223246968578b682c29ff5ef, State: Running, Role: LEADER
I20260619 06:27:46.020313 4042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789974ee223246968578b682c29ff5ef" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45719 } }
I20260619 06:27:46.020867 4037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:46.023269 4044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789974ee223246968578b682c29ff5ef. Latest consensus state: current_term: 1 leader_uuid: "789974ee223246968578b682c29ff5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789974ee223246968578b682c29ff5ef" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45719 } } }
I20260619 06:27:46.023144 4043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "789974ee223246968578b682c29ff5ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789974ee223246968578b682c29ff5ef" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45719 } } }
I20260619 06:27:46.023670 4043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:46.023577 4044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:46.025681 4051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:46.029783 4051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:46.037580 4051 catalog_manager.cc:1347] Generated new cluster ID: 84a6f28b6c374d71abe00689ec214c8f
I20260619 06:27:46.037675 4051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:46.055142 4051 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:46.056972 4051 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:46.066269 4051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 789974ee223246968578b682c29ff5ef: Generated new TSK 0
I20260619 06:27:46.067011 4051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:46.115046 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45719
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:46.439168 4063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:46.439440 4063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:46.439673 4063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:46.446831 4063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:46.446940 4063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:46.447151 4063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:46.456545 4063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:46.457800 4063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:46.459049 4063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:46.471807 4068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:46.471872 4069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:46.471833 4071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:46.473137 4063 server_base.cc:1034] running on GCE node
I20260619 06:27:46.473704 4063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:46.474505 4063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:46.475695 4063 hybrid_clock.cc:648] HybridClock initialized: now 1781850466475647 us; error 54 us; skew 500 ppm
I20260619 06:27:46.475980 4063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:46 dist-test-slave-4t1f krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:46.479007 4063 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:46.480477 4063 webserver.cc:460] Webserver started at http://127.0.97.65:37595/ using document root <none> and password file <none>
I20260619 06:27:46.481173 4063 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:46.481338 4063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:46.481630 4063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:46.484411 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "b61152cec01d4c6c88516ed4c079aa09"
format_stamp: "Formatted at 2026-06-19 06:27:46 on dist-test-slave-4t1f"
server_key: "125c0be67feda08d04ceb03282f829ab"
server_key_iv: "85d53adf229960834d7011ff275a1e48"
server_key_version: "encryptionkey@0"
I20260619 06:27:46.485231 4063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "b61152cec01d4c6c88516ed4c079aa09"
format_stamp: "Formatted at 2026-06-19 06:27:46 on dist-test-slave-4t1f"
server_key: "125c0be67feda08d04ceb03282f829ab"
server_key_iv: "85d53adf229960834d7011ff275a1e48"
server_key_version: "encryptionkey@0"
I20260619 06:27:46.490132 4063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260619 06:27:46.493471 4078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:46.494652 4063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:27:46.494812 4063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "b61152cec01d4c6c88516ed4c079aa09"
format_stamp: "Formatted at 2026-06-19 06:27:46 on dist-test-slave-4t1f"
server_key: "125c0be67feda08d04ceb03282f829ab"
server_key_iv: "85d53adf229960834d7011ff275a1e48"
server_key_version: "encryptionkey@0"
I20260619 06:27:46.494997 4063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:46.556903 4063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:46.561156 4063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:46.561439 4063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:46.562412 4063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:46.563653 4063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:46.563767 4063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:46.563910 4063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:46.564000 4063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:46.597242 4063 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:37677
I20260619 06:27:46.597285 4190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:37677 every 8 connection(s)
I20260619 06:27:46.598505 4063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:46.602921 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 4063
I20260619 06:27:46.603109 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:46.603724 389 external_mini_cluster.cc:1428] Setting key 387621cc55c78aa72ee49a18a8d20381
Jun 19 06:27:46 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:46.617705 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:46.600920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:51757 (local address 127.0.97.126:45719)
0619 06:27:46.601365 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:46.601369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:46.602290 (+ 921us) server_negotiation.cc:407] Connection header received
0619 06:27:46.604171 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:46.604176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:46.604268 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:46.604409 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:46.606059 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:46.606769 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:46.608269 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:46.608646 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:46.608775 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:46.612309 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:46.612314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:46.612317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:46.612345 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:46.614844 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:46.615531 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:46.615541 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:46.615544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:46.615650 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:46.616069 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:46.616073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:46.616077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:46.616328 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:46.616472 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:46.617162 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 06:27:46.617365 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":257,"threads_started":1}
I20260619 06:27:46.618654 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:46.601501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45719 (local address 127.0.97.65:51757)
0619 06:27:46.602146 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:46.602180 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:46.603893 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:46.604638 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:46.604647 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:46.605392 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:46.605906 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:46.605921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:46.606931 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:46.606935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:46.608113 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:46.608124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:46.608850 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:46.608858 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:46.609165 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:46.609834 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:46.609855 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:46.612076 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:46.614995 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:46.615004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:46.615020 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:46.615406 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:46.615766 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:46.615771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:46.615775 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:46.615954 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:46.616460 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:46.616467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:46.616776 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 06:27:46.617031 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 06:27:46.617521 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":255,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260619 06:27:46.620254 4191 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45719
I20260619 06:27:46.620522 4191 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:46.621280 4191 heartbeater.cc:511] Master 127.0.97.126:45719 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260619 06:27:46.623427 4002 ts_manager.cc:194] Registered new tserver with Master: b61152cec01d4c6c88516ed4c079aa09 (127.0.97.65:37677)
I20260619 06:27:46.625675 4002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:51757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:27:46.644115 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45719
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:46.950594 4197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:46.950840 4197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:46.951022 4197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:46.958005 4197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:46.958098 4197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:46.958263 4197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:46.967433 4197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:46.968487 4197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:46.969774 4197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:46.982347 4203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:46.982424 4202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:46.982398 4205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:46.983209 4197 server_base.cc:1034] running on GCE node
I20260619 06:27:46.983736 4197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:46.984535 4197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:46.985719 4197 hybrid_clock.cc:648] HybridClock initialized: now 1781850466985664 us; error 53 us; skew 500 ppm
I20260619 06:27:46.986017 4197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:46 dist-test-slave-4t1f krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:46.988907 4197 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:46.990281 4197 webserver.cc:460] Webserver started at http://127.0.97.66:35907/ using document root <none> and password file <none>
I20260619 06:27:46.990867 4197 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:46.990959 4197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:46.991195 4197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:46.994064 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "eb170536c8c9408399e2ea31e9cbc1aa"
format_stamp: "Formatted at 2026-06-19 06:27:46 on dist-test-slave-4t1f"
server_key: "b704a939c3cc25a32df9521126be4b99"
server_key_iv: "2739a17e2c8a6fbb28fd80799c73859b"
server_key_version: "encryptionkey@0"
I20260619 06:27:46.994753 4197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "eb170536c8c9408399e2ea31e9cbc1aa"
format_stamp: "Formatted at 2026-06-19 06:27:46 on dist-test-slave-4t1f"
server_key: "b704a939c3cc25a32df9521126be4b99"
server_key_iv: "2739a17e2c8a6fbb28fd80799c73859b"
server_key_version: "encryptionkey@0"
I20260619 06:27:46.999670 4197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260619 06:27:47.002902 4212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:47.004101 4197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:27:47.004253 4197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "eb170536c8c9408399e2ea31e9cbc1aa"
format_stamp: "Formatted at 2026-06-19 06:27:46 on dist-test-slave-4t1f"
server_key: "b704a939c3cc25a32df9521126be4b99"
server_key_iv: "2739a17e2c8a6fbb28fd80799c73859b"
server_key_version: "encryptionkey@0"
I20260619 06:27:47.004446 4197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:47.019500 4197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:47.023423 4197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:47.023701 4197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:47.024575 4197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:47.025961 4197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:47.026036 4197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:47.026223 4197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:47.026283 4197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:47.059006 4197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43233
I20260619 06:27:47.059051 4324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43233 every 8 connection(s)
I20260619 06:27:47.060258 4197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:47.069991 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 4197
I20260619 06:27:47.070164 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:47.070675 389 external_mini_cluster.cc:1428] Setting key 9d2e8313e9e60f8907d3783b0c9461b3
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:47.079766 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.062852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58283 (local address 127.0.97.126:45719)
0619 06:27:47.063044 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:47.063049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:47.064178 (+ 1129us) server_negotiation.cc:407] Connection header received
0619 06:27:47.065919 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:47.065923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:47.065999 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:47.066091 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:47.067498 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.068198 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.069875 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.070394 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.070488 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.074316 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:47.074320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:47.074323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:47.074352 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:47.076865 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.077606 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.077617 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.077621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.077735 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.078166 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.078176 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.078179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.078420 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:47.078571 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:47.079229 (+ 658us) server_negotiation.cc:299] Negotiation successful
0619 06:27:47.079441 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:27:47.080698 4327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.063324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45719 (local address 127.0.97.66:58283)
0619 06:27:47.064016 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:47.064059 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:47.065674 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:47.066284 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:47.066291 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:47.066836 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:47.067353 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:47.067368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.068345 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.068348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:47.069715 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:47.069732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.070610 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.070617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:47.070912 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.071703 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:47.071733 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:47.074106 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:47.077007 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:47.077017 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:47.077034 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:47.077484 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:47.077844 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:47.077851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:47.077857 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:47.078042 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:47.078546 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:47.078553 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:47.078883 (+ 330us) client_negotiation.cc:769] Sending connection context
0619 06:27:47.079133 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 06:27:47.079629 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":299,"threads_started":1}
I20260619 06:27:47.082314 4325 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45719
I20260619 06:27:47.082609 4325 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:47.083281 4325 heartbeater.cc:511] Master 127.0.97.126:45719 requested a full tablet report, sending...
I20260619 06:27:47.084796 4002 ts_manager.cc:194] Registered new tserver with Master: eb170536c8c9408399e2ea31e9cbc1aa (127.0.97.66:43233)
I20260619 06:27:47.086071 4002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:58283
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:47.113740 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45719
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:47.424165 4330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:47.424477 4330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:47.424664 4330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:47.432138 4330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:47.432307 4330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:47.432492 4330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:47.442072 4330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:47.443296 4330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:47.444607 4330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:47.457484 4336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:47.457487 4338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:47.457942 4335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:47.458087 4330 server_base.cc:1034] running on GCE node
I20260619 06:27:47.458844 4330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:47.459734 4330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:47.460935 4330 hybrid_clock.cc:648] HybridClock initialized: now 1781850467460895 us; error 25 us; skew 500 ppm
I20260619 06:27:47.461295 4330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:47.464341 4330 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:47.465816 4330 webserver.cc:460] Webserver started at http://127.0.97.67:38551/ using document root <none> and password file <none>
I20260619 06:27:47.466475 4330 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:47.466626 4330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:47.466917 4330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:47.469787 4330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "bf47fa071b9e4a4abe0b64d9c5b9d7fc"
format_stamp: "Formatted at 2026-06-19 06:27:47 on dist-test-slave-4t1f"
server_key: "1e771efab227f4cc22a0ca0b53267f0b"
server_key_iv: "fea2789dc76a0e744ee91efad57f41fd"
server_key_version: "encryptionkey@0"
I20260619 06:27:47.470515 4330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "bf47fa071b9e4a4abe0b64d9c5b9d7fc"
format_stamp: "Formatted at 2026-06-19 06:27:47 on dist-test-slave-4t1f"
server_key: "1e771efab227f4cc22a0ca0b53267f0b"
server_key_iv: "fea2789dc76a0e744ee91efad57f41fd"
server_key_version: "encryptionkey@0"
I20260619 06:27:47.475148 4330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:47.478288 4345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:47.479487 4330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:27:47.479650 4330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "bf47fa071b9e4a4abe0b64d9c5b9d7fc"
format_stamp: "Formatted at 2026-06-19 06:27:47 on dist-test-slave-4t1f"
server_key: "1e771efab227f4cc22a0ca0b53267f0b"
server_key_iv: "fea2789dc76a0e744ee91efad57f41fd"
server_key_version: "encryptionkey@0"
I20260619 06:27:47.479843 4330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:47.492871 4330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:47.497123 4330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:47.497406 4330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:47.498346 4330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:47.499672 4330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:47.499790 4330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:47.499931 4330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:47.500006 4330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:47.532128 4330 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:42491
I20260619 06:27:47.532171 4457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:42491 every 8 connection(s)
I20260619 06:27:47.533380 4330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:47.539898 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 4330
I20260619 06:27:47.540073 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:47.540480 389 external_mini_cluster.cc:1428] Setting key 345d34d0980ddee6088ae021790c5521
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850467, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:47.551913 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.535723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:42143 (local address 127.0.97.126:45719)
0619 06:27:47.535893 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:47.535897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:47.537019 (+ 1122us) server_negotiation.cc:407] Connection header received
0619 06:27:47.538883 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:47.538889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:47.539004 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:47.539113 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:47.540617 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.541565 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.543032 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.543458 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.543582 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.546787 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:47.546791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:47.546795 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:47.546823 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:47.549025 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.549761 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.549768 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.549772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.549902 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.550297 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.550303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.550306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.550550 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:47.550697 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:47.551315 (+ 618us) server_negotiation.cc:299] Negotiation successful
0619 06:27:47.551516 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:27:47.552759 4460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.536147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45719 (local address 127.0.97.67:42143)
0619 06:27:47.536858 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:47.536904 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:47.538611 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:47.539336 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:47.539343 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:47.539850 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:47.540459 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:47.540475 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.541699 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.541706 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:47.542883 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:47.542894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.543657 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.543666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:47.543910 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.544539 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:47.544560 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:47.546591 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:47.549222 (+ 2631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:47.549230 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:47.549243 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:47.549645 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:47.550006 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:47.550012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:47.550016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:47.550183 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:47.550671 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:47.550678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:47.550980 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 06:27:47.551226 (+ 246us) client_negotiation.cc:240] Negotiation successful
0619 06:27:47.551717 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":305,"threads_started":1}
I20260619 06:27:47.554118 4458 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45719
I20260619 06:27:47.554417 4458 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:47.555262 4458 heartbeater.cc:511] Master 127.0.97.126:45719 requested a full tablet report, sending...
I20260619 06:27:47.556526 4002 ts_manager.cc:194] Registered new tserver with Master: bf47fa071b9e4a4abe0b64d9c5b9d7fc (127.0.97.67:42491)
I20260619 06:27:47.557293 4002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:42143
I20260619 06:27:47.567239 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:47.618906 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.599768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41212 (local address 127.0.97.65:37677)
0619 06:27:47.605525 (+ 5757us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:47.605532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:47.605563 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:27:47.605614 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:47.605620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:47.605808 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:47.605946 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:47.607221 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.608315 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.609628 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.610154 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.610262 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.613813 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:47.613822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:47.613835 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:47.613861 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:47.616714 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.617289 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.617295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.617298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.617405 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.617700 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.617703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.617705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.617992 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:47.618204 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:47.618379 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 06:27:47.618541 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5637,"thread_start_us":5454,"threads_started":1}
I20260619 06:27:47.628688 4191 heartbeater.cc:503] Master 127.0.97.126:45719 was elected leader, sending a full tablet report...
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:47.635839 4193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.622766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.0.97.126:45719)
0619 06:27:47.622967 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:47.622973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:47.623137 (+ 164us) server_negotiation.cc:407] Connection header received
0619 06:27:47.623353 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:47.623359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:47.623425 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:47.623522 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:47.624661 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.625559 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.626780 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.627157 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.627331 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.631260 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:47.631264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:47.631267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:47.631295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:47.633770 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.634274 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.634296 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.634298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.634406 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.634688 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.634691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.634693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.634970 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:47.635130 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:47.635354 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 06:27:47.635490 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 06:27:47.636292 4002 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36290
I20260619 06:27:47.637226 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 4063
I20260619 06:27:47.648159 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.637945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41214 (local address 127.0.97.65:37677)
0619 06:27:47.638147 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:47.638152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:47.638316 (+ 164us) server_negotiation.cc:407] Connection header received
0619 06:27:47.638426 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:47.638430 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:47.638491 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:47.638564 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:47.639604 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.640241 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.641348 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.641730 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.641818 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.643705 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:47.643710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:47.643713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:47.643738 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:47.646214 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.646720 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.646724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.646726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.646826 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.647100 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.647103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.647105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.647315 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:47.647469 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:47.647690 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 06:27:47.647827 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:27:47.648669 4186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:47.729416 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 4063
I20260619 06:27:47.744855 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 4197
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:27:47.759887 4470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.745699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.0.97.66:43233)
0619 06:27:47.746199 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:47.746206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:47.746230 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:27:47.746291 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:47.746297 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:47.746496 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:47.746669 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:47.747787 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.748768 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.750091 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.750610 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.750743 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.754711 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:47.754721 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:47.754733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:47.754760 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:47.757309 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.757911 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.757921 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.757929 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.758042 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.758413 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.758422 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.758425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.758719 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:47.759013 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:47.759300 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 06:27:47.759550 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":277,"threads_started":1}
I20260619 06:27:47.761128 4320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:47.841477 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 4197
I20260619 06:27:47.856638 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 4330
Jun 19 06:27:47 dist-test-slave-4t1f krb5kdc[3957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:27:47.871393 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:47.857669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46660 (local address 127.0.97.67:42491)
0619 06:27:47.858277 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:47.858283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:47.858309 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:27:47.858364 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:47.858370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:47.858587 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:47.858706 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:47.859879 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.860843 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.862052 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:47.862569 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:47.862662 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:47.866693 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:47.866703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:47.866717 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:47.866745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:47.869296 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.869840 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.869846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.869848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.869957 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:47.870252 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:47.870255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:47.870257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:47.870537 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:47.870748 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:47.870944 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 06:27:47.871114 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":324,"threads_started":1}
I20260619 06:27:47.872629 4453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:47.954284 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 4330
I20260619 06:27:47.969266 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 3970
I20260619 06:27:47.970376 4032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:48.068972 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 3970
2026-06-19T06:27:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3787 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:48 dist-test-slave-4t1f krb5kdc[4477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:48 dist-test-slave-4t1f krb5kdc[4477](info): set up 2 sockets
Jun 19 06:27:48 dist-test-slave-4t1f krb5kdc[4477](info): commencing operation
krb5kdc: starting...
W20260619 06:27:49.165433 389 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:27:49 dist-test-slave-4t1f krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:27:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:27:49.348436 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38059
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:49.650991 4490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:49.651299 4490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:49.651460 4490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:49.658139 4490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:49.658270 4490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:49.658375 4490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:49.658478 4490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:49.658562 4490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:49.667933 4490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:49.669154 4490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:49.670465 4490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:49.681797 4496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:49.681804 4495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:49.682077 4498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:49.682552 4490 server_base.cc:1034] running on GCE node
I20260619 06:27:49.683357 4490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:49.684718 4490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:49.685945 4490 hybrid_clock.cc:648] HybridClock initialized: now 1781850469685896 us; error 49 us; skew 500 ppm
I20260619 06:27:49.686297 4490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:49 dist-test-slave-4t1f krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850469, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:49.689292 4490 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:27:49.690366 4490 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:49.690491 4490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:49.690804 4490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:49.693616 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "5329fc7351184460a092380b1d44ddbb"
format_stamp: "Formatted at 2026-06-19 06:27:49 on dist-test-slave-4t1f"
server_key: "ce0252cfc01c3fc2f4bc6cf172f44b86"
server_key_iv: "978408aa5a178ccf0456dd5f203e69e9"
server_key_version: "encryptionkey@0"
I20260619 06:27:49.694430 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "5329fc7351184460a092380b1d44ddbb"
format_stamp: "Formatted at 2026-06-19 06:27:49 on dist-test-slave-4t1f"
server_key: "ce0252cfc01c3fc2f4bc6cf172f44b86"
server_key_iv: "978408aa5a178ccf0456dd5f203e69e9"
server_key_version: "encryptionkey@0"
I20260619 06:27:49.699105 4490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:49.702248 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:49.703393 4490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:27:49.703557 4490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "5329fc7351184460a092380b1d44ddbb"
format_stamp: "Formatted at 2026-06-19 06:27:49 on dist-test-slave-4t1f"
server_key: "ce0252cfc01c3fc2f4bc6cf172f44b86"
server_key_iv: "978408aa5a178ccf0456dd5f203e69e9"
server_key_version: "encryptionkey@0"
I20260619 06:27:49.703761 4490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:49.725107 4490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:49.729210 4490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:49.729417 4490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:49.746995 4490 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38059
I20260619 06:27:49.747015 4555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38059 every 8 connection(s)
I20260619 06:27:49.747816 4490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:49.751197 4556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:49.754951 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 4490
I20260619 06:27:49.755136 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:49.755787 389 external_mini_cluster.cc:1428] Setting key e42878e5ea3615e8de9646db58de61ac
I20260619 06:27:49.762943 4556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb: Bootstrap starting.
I20260619 06:27:49.766376 4556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:49.767601 4556 log.cc:824] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:49.770422 4556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb: No bootstrap required, opened a new log
Jun 19 06:27:49 dist-test-slave-4t1f krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:49.780006 4559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:49.757734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53624 (local address 127.0.97.126:38059)
0619 06:27:49.758484 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:49.758499 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:49.758550 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:27:49.761222 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:49.761248 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:49.761729 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:49.762129 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:49.763448 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:49.765508 (+ 2060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:49.766776 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:49.767809 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:49.767981 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:49.771793 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:49.771804 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:49.771823 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:49.771872 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:49.775776 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:49.776376 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:49.776384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:49.776392 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:49.776597 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:49.776918 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:49.776923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:49.776926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:49.777609 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:49.777884 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:49.778190 (+ 306us) server_negotiation.cc:299] Negotiation successful
0619 06:27:49.778640 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":292,"threads_started":1}
I20260619 06:27:49.780908 4556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5329fc7351184460a092380b1d44ddbb" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38059 } }
I20260619 06:27:49.781334 4556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:49.781442 4556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5329fc7351184460a092380b1d44ddbb, State: Initialized, Role: FOLLOWER
I20260619 06:27:49.782014 4556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5329fc7351184460a092380b1d44ddbb" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38059 } }
I20260619 06:27:49.782238 4556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:49.782387 4556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:49.782564 4556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:49.784788 4556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5329fc7351184460a092380b1d44ddbb" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38059 } }
I20260619 06:27:49.785367 4556 leader_election.cc:302] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5329fc7351184460a092380b1d44ddbb; no voters:
I20260619 06:27:49.785904 4556 leader_election.cc:288] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:49.786024 4561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:49.787119 4561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [term 1 LEADER]: Becoming Leader. State: Replica: 5329fc7351184460a092380b1d44ddbb, State: Running, Role: LEADER
I20260619 06:27:49.787667 4561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5329fc7351184460a092380b1d44ddbb" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38059 } }
I20260619 06:27:49.787981 4556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:49.790355 4563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5329fc7351184460a092380b1d44ddbb. Latest consensus state: current_term: 1 leader_uuid: "5329fc7351184460a092380b1d44ddbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5329fc7351184460a092380b1d44ddbb" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38059 } } }
I20260619 06:27:49.790669 4563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:49.790446 4562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5329fc7351184460a092380b1d44ddbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5329fc7351184460a092380b1d44ddbb" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38059 } } }
I20260619 06:27:49.790845 4562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:49.792222 4567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:49.797001 4567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:49.804755 4567 catalog_manager.cc:1347] Generated new cluster ID: 73e8f0f80a1c4d12ac24bc63fb76282b
I20260619 06:27:49.804869 4567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:49.821827 4567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:49.823539 4567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:49.843137 4567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb: Generated new TSK 0
I20260619 06:27:49.844105 4567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:27:49.886775 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38059
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:50.203575 4582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:50.203872 4582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:50.203971 4582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:50.211161 4582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:50.211272 4582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:50.211398 4582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:50.220878 4582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:50.222061 4582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:50.223263 4582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:50.235164 4590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:50.235229 4588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:50.235205 4587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:50.235771 4582 server_base.cc:1034] running on GCE node
I20260619 06:27:50.236299 4582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:50.237111 4582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:50.238328 4582 hybrid_clock.cc:648] HybridClock initialized: now 1781850470238267 us; error 48 us; skew 500 ppm
I20260619 06:27:50.238637 4582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:50 dist-test-slave-4t1f krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:50.241765 4582 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:27:50.242786 4582 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:50.242893 4582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:50.243207 4582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:50.245940 4582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "9d755efcea9f40319275df2e22fda3d8"
format_stamp: "Formatted at 2026-06-19 06:27:50 on dist-test-slave-4t1f"
server_key: "5cfbdfd38df223231c58dbb9a02cd2db"
server_key_iv: "a53bf540f2af4d0c148d261919bc9229"
server_key_version: "encryptionkey@0"
I20260619 06:27:50.246698 4582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "9d755efcea9f40319275df2e22fda3d8"
format_stamp: "Formatted at 2026-06-19 06:27:50 on dist-test-slave-4t1f"
server_key: "5cfbdfd38df223231c58dbb9a02cd2db"
server_key_iv: "a53bf540f2af4d0c148d261919bc9229"
server_key_version: "encryptionkey@0"
I20260619 06:27:50.251262 4582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:27:50.254467 4596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:50.255662 4582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:27:50.255815 4582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "9d755efcea9f40319275df2e22fda3d8"
format_stamp: "Formatted at 2026-06-19 06:27:50 on dist-test-slave-4t1f"
server_key: "5cfbdfd38df223231c58dbb9a02cd2db"
server_key_iv: "a53bf540f2af4d0c148d261919bc9229"
server_key_version: "encryptionkey@0"
I20260619 06:27:50.256028 4582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:50.285882 4582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:50.289975 4582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:50.290244 4582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:50.291144 4582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:50.292387 4582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:50.292472 4582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:50.292595 4582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:50.292667 4582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:50.324560 4582 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:43071
I20260619 06:27:50.324641 4708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:43071 every 8 connection(s)
I20260619 06:27:50.325444 4582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:50.334316 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 4582
I20260619 06:27:50.334506 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:50.335309 389 external_mini_cluster.cc:1428] Setting key 76d1f5f9a7d809093672f1938a06f8f1
Jun 19 06:27:50 dist-test-slave-4t1f krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:50.344807 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:50.327802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:46447 (local address 127.0.97.126:38059)
0619 06:27:50.328204 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:50.328210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:50.329296 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 06:27:50.330990 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:50.330994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:50.331054 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:50.331143 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:50.332460 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.333220 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.334750 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.335393 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.335545 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:50.339363 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:50.339368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:50.339371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:50.339401 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:50.341917 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:50.342620 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:50.342627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:50.342630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:50.342758 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:50.343171 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:50.343178 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:50.343181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:50.343465 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:50.343607 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:50.344226 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 06:27:50.344462 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":188,"threads_started":1}
I20260619 06:27:50.345834 4712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:50.328441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38059 (local address 127.0.97.65:46447)
0619 06:27:50.329140 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:50.329192 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:50.330759 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:50.331321 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:50.331328 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:50.331805 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:50.332316 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:50.332334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.333337 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.333341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:50.334606 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:50.334621 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.335595 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.335599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:50.335769 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:50.336409 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:50.336431 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:50.339166 (+ 2735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:50.342052 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:50.342060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:50.342075 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:50.342496 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:50.342867 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:50.342874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:50.342879 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:50.343052 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:50.343584 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:50.343592 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:50.343893 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 06:27:50.344155 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 06:27:50.344632 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":263,"threads_started":1}
I20260619 06:27:50.347329 4709 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38059
I20260619 06:27:50.347601 4709 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:50.348177 4709 heartbeater.cc:511] Master 127.0.97.126:38059 requested a full tablet report, sending...
I20260619 06:27:50.350106 4521 ts_manager.cc:194] Registered new tserver with Master: 9d755efcea9f40319275df2e22fda3d8 (127.0.97.65:43071)
I20260619 06:27:50.351397 4521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:46447
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:27:50.375952 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38059
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:50.690649 4715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:50.690894 4715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:50.691047 4715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:50.698036 4715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:50.698128 4715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:50.698294 4715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:50.707373 4715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:50.708544 4715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:50.709795 4715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:50.722004 4720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:50.721974 4723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:50.722050 4721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:50.722743 4715 server_base.cc:1034] running on GCE node
I20260619 06:27:50.723311 4715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:50.724103 4715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:50.725304 4715 hybrid_clock.cc:648] HybridClock initialized: now 1781850470725266 us; error 30 us; skew 500 ppm
I20260619 06:27:50.725603 4715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:50 dist-test-slave-4t1f krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:50.728389 4715 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:27:50.729477 4715 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:50.729585 4715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:50.729833 4715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:50.732522 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "98abf207651f4584afffdc981b5ea706"
format_stamp: "Formatted at 2026-06-19 06:27:50 on dist-test-slave-4t1f"
server_key: "55e16d91d3a96907d9e9f869c96d4c58"
server_key_iv: "ee7ab59bfff9669cc03aaa72c1b7be67"
server_key_version: "encryptionkey@0"
I20260619 06:27:50.733376 4715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "98abf207651f4584afffdc981b5ea706"
format_stamp: "Formatted at 2026-06-19 06:27:50 on dist-test-slave-4t1f"
server_key: "55e16d91d3a96907d9e9f869c96d4c58"
server_key_iv: "ee7ab59bfff9669cc03aaa72c1b7be67"
server_key_version: "encryptionkey@0"
I20260619 06:27:50.737910 4715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:27:50.741027 4729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:50.742245 4715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:27:50.742420 4715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "98abf207651f4584afffdc981b5ea706"
format_stamp: "Formatted at 2026-06-19 06:27:50 on dist-test-slave-4t1f"
server_key: "55e16d91d3a96907d9e9f869c96d4c58"
server_key_iv: "ee7ab59bfff9669cc03aaa72c1b7be67"
server_key_version: "encryptionkey@0"
I20260619 06:27:50.742630 4715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:50.764490 4715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:50.768122 4715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:50.768371 4715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:50.769302 4715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:50.770509 4715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:50.770627 4715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:50.770752 4715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:50.770843 4715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:50.802420 4715 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:45139
I20260619 06:27:50.802456 4841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:45139 every 8 connection(s)
I20260619 06:27:50.803217 4715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:50.812480 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 4715
I20260619 06:27:50.812665 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:50.813163 389 external_mini_cluster.cc:1428] Setting key 7fcb47bbf983432df3c3d243e3476672
Jun 19 06:27:50 dist-test-slave-4t1f krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850470, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:50.822328 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:50.805762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46365 (local address 127.0.97.126:38059)
0619 06:27:50.805955 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:50.805961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:50.807105 (+ 1144us) server_negotiation.cc:407] Connection header received
0619 06:27:50.808927 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:50.808932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:50.808990 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:50.809134 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:50.810483 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.811175 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.812620 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.813109 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.813241 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:50.816984 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:50.816989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:50.816993 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:50.817021 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:50.819517 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:50.820187 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:50.820194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:50.820197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:50.820325 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:50.820745 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:50.820750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:50.820753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:50.820989 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:50.821158 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:50.821783 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 06:27:50.821990 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:27:50.823254 4844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:50.806276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38059 (local address 127.0.97.66:46365)
0619 06:27:50.806960 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:50.807001 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:50.808690 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:50.809299 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:50.809307 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:50.809806 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:50.810351 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:50.810367 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.811289 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.811292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:50.812465 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:50.812476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:50.813345 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:50.813352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:50.813569 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:50.814367 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:50.814393 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:50.816781 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:50.819650 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:50.819659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:50.819675 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:50.820072 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:50.820435 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:50.820440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:50.820444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:50.820626 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:50.821143 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:50.821150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:50.821451 (+ 301us) client_negotiation.cc:769] Sending connection context
0619 06:27:50.821712 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 06:27:50.822206 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":260,"threads_started":1}
I20260619 06:27:50.824630 4842 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38059
I20260619 06:27:50.824915 4842 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:50.825587 4842 heartbeater.cc:511] Master 127.0.97.126:38059 requested a full tablet report, sending...
I20260619 06:27:50.826998 4521 ts_manager.cc:194] Registered new tserver with Master: 98abf207651f4584afffdc981b5ea706 (127.0.97.66:45139)
I20260619 06:27:50.828284 4521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:46365
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:27:50.853528 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38059
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:27:51.158354 4847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:51.158641 4847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:51.158780 4847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:51.165621 4847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:51.165750 4847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:51.165925 4847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:51.174832 4847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:51.175971 4847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:51.177199 4847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:51.189368 4852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:51.189288 4853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:51.189316 4855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:51.190168 4847 server_base.cc:1034] running on GCE node
I20260619 06:27:51.190749 4847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:51.191576 4847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:51.192771 4847 hybrid_clock.cc:648] HybridClock initialized: now 1781850471192738 us; error 32 us; skew 500 ppm
I20260619 06:27:51.193020 4847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:27:51 dist-test-slave-4t1f krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:27:51.199229 4847 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:27:51.200266 4847 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:51.200348 4847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:51.200706 4847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:51.203472 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "2afb7f4029694bda9e5281f91a444827"
format_stamp: "Formatted at 2026-06-19 06:27:51 on dist-test-slave-4t1f"
server_key: "eb5ba4127eae00451ccf80c22d676c1f"
server_key_iv: "00d7d8f1947ce5ac0dc9e55ad84f3ad7"
server_key_version: "encryptionkey@0"
I20260619 06:27:51.204254 4847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "2afb7f4029694bda9e5281f91a444827"
format_stamp: "Formatted at 2026-06-19 06:27:51 on dist-test-slave-4t1f"
server_key: "eb5ba4127eae00451ccf80c22d676c1f"
server_key_iv: "00d7d8f1947ce5ac0dc9e55ad84f3ad7"
server_key_version: "encryptionkey@0"
I20260619 06:27:51.208920 4847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 06:27:51.212013 4861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:51.213335 4847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:51.213500 4847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "2afb7f4029694bda9e5281f91a444827"
format_stamp: "Formatted at 2026-06-19 06:27:51 on dist-test-slave-4t1f"
server_key: "eb5ba4127eae00451ccf80c22d676c1f"
server_key_iv: "00d7d8f1947ce5ac0dc9e55ad84f3ad7"
server_key_version: "encryptionkey@0"
I20260619 06:27:51.213696 4847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:51.243235 4847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:51.247480 4847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:51.247745 4847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:51.248615 4847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:51.249833 4847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:51.249938 4847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:51.250062 4847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:51.250156 4847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:51.281553 4847 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:43877
I20260619 06:27:51.281667 4973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:43877 every 8 connection(s)
I20260619 06:27:51.282367 4847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:51.290175 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 4847
I20260619 06:27:51.290395 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:51.290938 389 external_mini_cluster.cc:1428] Setting key c1718e3854842a6f36e5aae8074d4635
Jun 19 06:27:51 dist-test-slave-4t1f krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850471, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:27:51.300746 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.284585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51769 (local address 127.0.97.126:38059)
0619 06:27:51.284784 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.284791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.285975 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 06:27:51.287658 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.287662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.287728 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.287819 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:51.289330 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.290052 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.292136 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.292547 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.292635 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.296067 (+ 3432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:51.296071 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:51.296074 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:51.296102 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:51.298442 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:51.299039 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:51.299042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:51.299045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:51.299156 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:51.299514 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:51.299517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:51.299519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:51.299736 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:51.299832 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.300385 (+ 553us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.300521 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:27:51.301651 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.285173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38059 (local address 127.0.97.67:51769)
0619 06:27:51.285823 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:51.285859 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:51.287425 (+ 1566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:51.288069 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:51.288077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:51.288547 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:51.289196 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.289214 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.290203 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.290209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.291987 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.292000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.292799 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.292810 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.293117 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.293838 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:27:51.293876 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:51.295890 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:51.298553 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:51.298558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:51.298569 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:51.298941 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:51.299240 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:27:51.299243 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:27:51.299246 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:27:51.299402 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:27:51.299829 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:51.299834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:51.300089 (+ 255us) client_negotiation.cc:769] Sending connection context
0619 06:27:51.300305 (+ 216us) client_negotiation.cc:240] Negotiation successful
0619 06:27:51.300621 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":260,"threads_started":1}
I20260619 06:27:51.303013 4974 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38059
I20260619 06:27:51.303282 4974 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:51.303862 4974 heartbeater.cc:511] Master 127.0.97.126:38059 requested a full tablet report, sending...
I20260619 06:27:51.305349 4521 ts_manager.cc:194] Registered new tserver with Master: 2afb7f4029694bda9e5281f91a444827 (127.0.97.67:43877)
I20260619 06:27:51.306169 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:27:51.306396 4521 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:51769
I20260619 06:27:51.321849 4711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.310607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53628 (local address 127.0.97.126:38059)
0619 06:27:51.310743 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.310747 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.311189 (+ 442us) server_negotiation.cc:407] Connection header received
0619 06:27:51.311307 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.311310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.311362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.311435 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:51.312692 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.313407 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.314527 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.314903 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.314986 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.317014 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:51.317019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:51.317024 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:51.317119 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:51.319352 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:51.319993 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:51.320000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:51.320003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:51.320130 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:51.320485 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:51.320493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:51.320498 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:51.320772 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:51.321042 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.321353 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.321540 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:27:51.325685 4521 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:51.333945 4521 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:51.354257 4709 heartbeater.cc:503] Master 127.0.97.126:38059 was elected leader, sending a full tablet report...
I20260619 06:27:51.366791 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.344808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33696)
0619 06:27:51.357237 (+ 12429us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:51.357281 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:51.357438 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:51.358695 (+ 1257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:51.358706 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:51.358748 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:51.359265 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.359282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.361427 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.361432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.363894 (+ 2462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.363908 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.365774 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.365781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.365991 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.366046 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:27:51.366167 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 06:27:51.366336 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12168,"thread_start_us":15138,"threads_started":1}
I20260619 06:27:51.367884 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.345021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.0.97.65:43071)
0619 06:27:51.357709 (+ 12688us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.357721 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.357758 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:27:51.357876 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.357887 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.358188 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.359484 (+ 1296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:51.359806 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.361296 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.364081 (+ 2785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.365568 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.365712 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.365923 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.367379 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.367556 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12454,"thread_start_us":13315,"threads_started":1}
I20260619 06:27:51.369364 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.345513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.0.97.67:43877)
0619 06:27:51.356519 (+ 11006us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.356532 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.359922 (+ 3390us) server_negotiation.cc:407] Connection header received
0619 06:27:51.361488 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.361496 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.361750 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.361910 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:51.362856 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.364361 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.366840 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.368102 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.368202 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.368355 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.368860 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.369017 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10787,"thread_start_us":11023,"threads_started":1}
I20260619 06:27:51.370116 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.345163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:43877 (local address 127.0.0.1:55234)
0619 06:27:51.359808 (+ 14645us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:51.359825 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:51.361280 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:51.362206 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:51.362211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:51.362234 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:51.362652 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.362663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.364518 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.364523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.366652 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.366667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.368512 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.368516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.368653 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.368688 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:27:51.369644 (+ 956us) client_negotiation.cc:240] Negotiation successful
0619 06:27:51.369773 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14532,"thread_start_us":11979,"threads_started":1}
I20260619 06:27:51.371296 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.346757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:45139 (local address 127.0.0.1:33316)
0619 06:27:51.363787 (+ 17030us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:51.363804 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:51.363914 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:51.364467 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:51.364472 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:51.364494 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:51.364897 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.364905 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.366466 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.366472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.368549 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.368563 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.370601 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.370606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.370782 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.370810 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:27:51.370902 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:27:51.371016 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16894,"thread_start_us":16753,"threads_started":1}
I20260619 06:27:51.371431 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.347425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.0.97.66:45139)
0619 06:27:51.363113 (+ 15688us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.363125 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.364034 (+ 909us) server_negotiation.cc:407] Connection header received
0619 06:27:51.364133 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.364138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.364301 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.364480 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:51.365081 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.366316 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.368718 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.370367 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.370501 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.370699 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.370950 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.371120 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15486,"thread_start_us":15710,"threads_started":1}
I20260619 06:27:51.371732 4644 tablet_service.cc:1467] Processing CreateTablet for tablet ee14b0c7fa254288a9f4607883142724 (DEFAULT_TABLE table=test-table [id=69a74e13de0d415fa4d6b449a2d7bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:51.373072 4644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee14b0c7fa254288a9f4607883142724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:51.373442 4909 tablet_service.cc:1467] Processing CreateTablet for tablet ee14b0c7fa254288a9f4607883142724 (DEFAULT_TABLE table=test-table [id=69a74e13de0d415fa4d6b449a2d7bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:51.374743 4909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee14b0c7fa254288a9f4607883142724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:51.374738 4777 tablet_service.cc:1467] Processing CreateTablet for tablet ee14b0c7fa254288a9f4607883142724 (DEFAULT_TABLE table=test-table [id=69a74e13de0d415fa4d6b449a2d7bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:51.375975 4777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee14b0c7fa254288a9f4607883142724. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:51.386065 4992 tablet_bootstrap.cc:492] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Bootstrap starting.
I20260619 06:27:51.388120 4993 tablet_bootstrap.cc:492] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: Bootstrap starting.
I20260619 06:27:51.389638 4992 tablet_bootstrap.cc:654] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:51.390885 4992 log.cc:824] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:51.391607 4993 tablet_bootstrap.cc:654] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:51.391691 4994 tablet_bootstrap.cc:492] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Bootstrap starting.
I20260619 06:27:51.392915 4993 log.cc:824] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:51.394850 4994 tablet_bootstrap.cc:654] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:51.395140 4992 tablet_bootstrap.cc:492] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: No bootstrap required, opened a new log
I20260619 06:27:51.395409 4992 ts_tablet_manager.cc:1397] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 06:27:51.395793 4993 tablet_bootstrap.cc:492] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: No bootstrap required, opened a new log
I20260619 06:27:51.395833 4994 log.cc:824] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:51.396059 4993 ts_tablet_manager.cc:1397] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:27:51.398222 4994 tablet_bootstrap.cc:492] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: No bootstrap required, opened a new log
I20260619 06:27:51.398461 4994 ts_tablet_manager.cc:1397] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:27:51.405462 4994 raft_consensus.cc:348] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.405810 4994 raft_consensus.cc:374] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:51.405931 4994 raft_consensus.cc:729] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98abf207651f4584afffdc981b5ea706, State: Initialized, Role: FOLLOWER
I20260619 06:27:51.406507 4994 consensus_queue.cc:260] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.406517 4993 raft_consensus.cc:348] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.406517 4992 raft_consensus.cc:348] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.406909 4993 raft_consensus.cc:374] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:51.406912 4992 raft_consensus.cc:374] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:51.407007 4992 raft_consensus.cc:729] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2afb7f4029694bda9e5281f91a444827, State: Initialized, Role: FOLLOWER
I20260619 06:27:51.407079 4993 raft_consensus.cc:729] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d755efcea9f40319275df2e22fda3d8, State: Initialized, Role: FOLLOWER
I20260619 06:27:51.407581 4842 heartbeater.cc:503] Master 127.0.97.126:38059 was elected leader, sending a full tablet report...
I20260619 06:27:51.407727 4993 consensus_queue.cc:260] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.407725 4992 consensus_queue.cc:260] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.408353 4994 ts_tablet_manager.cc:1428] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Time spent starting tablet: real 0.010s user 0.006s sys 0.005s
I20260619 06:27:51.409837 4974 heartbeater.cc:503] Master 127.0.97.126:38059 was elected leader, sending a full tablet report...
I20260619 06:27:51.410216 4993 ts_tablet_manager.cc:1428] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260619 06:27:51.411309 4992 ts_tablet_manager.cc:1428] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
W20260619 06:27:51.533800 4975 tablet.cc:2369] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:27:51.555018 4843 tablet.cc:2369] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:27:51.577556 4710 tablet.cc:2369] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:27:51.752338 4999 raft_consensus.cc:482] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:51.752624 4999 raft_consensus.cc:504] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.754568 4999 leader_election.cc:288] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2afb7f4029694bda9e5281f91a444827 (127.0.97.67:43877), 98abf207651f4584afffdc981b5ea706 (127.0.97.66:45139)
I20260619 06:27:51.762557 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.755114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:41371 (local address 127.0.97.67:43877)
0619 06:27:51.755423 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.755434 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.756461 (+ 1027us) server_negotiation.cc:407] Connection header received
0619 06:27:51.756616 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.756621 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.756690 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.756796 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:51.757473 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.758332 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.760473 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.761631 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.761757 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.761861 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.762173 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.762302 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260619 06:27:51.762650 5001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.754936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:43877 (local address 127.0.97.65:41371)
0619 06:27:51.756348 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:51.756370 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:51.756495 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:51.756830 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:51.756839 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:51.756859 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:51.757329 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.757337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.758464 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.758472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.760273 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.760286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.761833 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.761839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.762005 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.762038 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:27:51.762125 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:27:51.762244 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"mutex_wait_us":103,"thread_start_us":649,"threads_started":1}
I20260619 06:27:51.763238 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.755283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:35369 (local address 127.0.97.66:45139)
0619 06:27:51.755452 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.755457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.756016 (+ 559us) server_negotiation.cc:407] Connection header received
0619 06:27:51.756342 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.756348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.756439 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.756529 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:27:51.757244 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.758355 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.760922 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.762081 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.762220 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.762302 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.762842 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.763001 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:27:51.763891 5002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.754834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:45139 (local address 127.0.97.65:35369)
0619 06:27:51.755916 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:51.755938 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:51.756131 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:51.756637 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:51.756642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:51.756669 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:27:51.757099 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.757115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.758511 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.758516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.760767 (+ 2251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:51.760783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.762453 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.762459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:51.762640 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.762685 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:27:51.763420 (+ 735us) client_negotiation.cc:240] Negotiation successful
0619 06:27:51.763548 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":27,"thread_start_us":1203,"threads_started":1}
I20260619 06:27:51.763799 4929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee14b0c7fa254288a9f4607883142724" candidate_uuid: "9d755efcea9f40319275df2e22fda3d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afb7f4029694bda9e5281f91a444827" is_pre_election: true
I20260619 06:27:51.764343 4929 raft_consensus.cc:2454] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d755efcea9f40319275df2e22fda3d8 in term 0.
I20260619 06:27:51.764850 4797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee14b0c7fa254288a9f4607883142724" candidate_uuid: "9d755efcea9f40319275df2e22fda3d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98abf207651f4584afffdc981b5ea706" is_pre_election: true
I20260619 06:27:51.764957 4598 leader_election.cc:302] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2afb7f4029694bda9e5281f91a444827, 9d755efcea9f40319275df2e22fda3d8; no voters:
I20260619 06:27:51.765370 4999 raft_consensus.cc:2781] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:51.765313 4797 raft_consensus.cc:2454] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d755efcea9f40319275df2e22fda3d8 in term 0.
I20260619 06:27:51.765518 4999 raft_consensus.cc:482] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:51.765625 4999 raft_consensus.cc:3037] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:51.767686 4999 raft_consensus.cc:504] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.768307 4999 leader_election.cc:288] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [CANDIDATE]: Term 1 election: Requested vote from peers 2afb7f4029694bda9e5281f91a444827 (127.0.97.67:43877), 98abf207651f4584afffdc981b5ea706 (127.0.97.66:45139)
I20260619 06:27:51.768666 4929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee14b0c7fa254288a9f4607883142724" candidate_uuid: "9d755efcea9f40319275df2e22fda3d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2afb7f4029694bda9e5281f91a444827"
I20260619 06:27:51.768787 4797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee14b0c7fa254288a9f4607883142724" candidate_uuid: "9d755efcea9f40319275df2e22fda3d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "98abf207651f4584afffdc981b5ea706"
I20260619 06:27:51.768862 4929 raft_consensus.cc:3037] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:51.768980 4797 raft_consensus.cc:3037] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:51.771957 4929 raft_consensus.cc:2454] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d755efcea9f40319275df2e22fda3d8 in term 1.
I20260619 06:27:51.771958 4797 raft_consensus.cc:2454] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d755efcea9f40319275df2e22fda3d8 in term 1.
I20260619 06:27:51.772322 4598 leader_election.cc:302] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2afb7f4029694bda9e5281f91a444827, 98abf207651f4584afffdc981b5ea706, 9d755efcea9f40319275df2e22fda3d8; no voters:
I20260619 06:27:51.772605 4999 raft_consensus.cc:2781] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:51.773022 4999 raft_consensus.cc:686] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 1 LEADER]: Becoming Leader. State: Replica: 9d755efcea9f40319275df2e22fda3d8, State: Running, Role: LEADER
I20260619 06:27:51.773589 4999 consensus_queue.cc:237] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } }
I20260619 06:27:51.779093 4520 catalog_manager.cc:5495] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d755efcea9f40319275df2e22fda3d8 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "9d755efcea9f40319275df2e22fda3d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d755efcea9f40319275df2e22fda3d8" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20260619 06:27:51.824333 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.819106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.0.97.65:43071)
0619 06:27:51.819294 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.819298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.819470 (+ 172us) server_negotiation.cc:407] Connection header received
0619 06:27:51.819639 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.819642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.819713 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.819836 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:27:51.820119 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.820817 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.822659 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.823026 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.823114 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.823481 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:27:51.823927 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:27:51.824026 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.824098 (+ 72us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.824153 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:27:51.832010 4797 raft_consensus.cc:1261] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 1 FOLLOWER]: Refusing update from remote peer 9d755efcea9f40319275df2e22fda3d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:51.832010 4929 raft_consensus.cc:1261] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 1 FOLLOWER]: Refusing update from remote peer 9d755efcea9f40319275df2e22fda3d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:51.833002 5003 consensus_queue.cc:1035] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2afb7f4029694bda9e5281f91a444827" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 43877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:51.833393 4999 consensus_queue.cc:1035] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98abf207651f4584afffdc981b5ea706" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:51.844851 5008 mvcc.cc:205] Tried to move back new op lower bound from 7298459532613812224 to 7298459532388442112. Current Snapshot: MvccSnapshot[applied={T|T < 7298459532613812224}]
I20260619 06:27:51.854962 4520 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:51.855194 4520 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:51.858314 4520 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5329fc7351184460a092380b1d44ddbb: Sending DeleteTablet for 3 replicas of tablet ee14b0c7fa254288a9f4607883142724
I20260619 06:27:51.859407 4909 tablet_service.cc:1514] Processing DeleteTablet for tablet ee14b0c7fa254288a9f4607883142724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:51 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:55234
I20260619 06:27:51.859403 4644 tablet_service.cc:1514] Processing DeleteTablet for tablet ee14b0c7fa254288a9f4607883142724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:51 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:33696
I20260619 06:27:51.859730 4777 tablet_service.cc:1514] Processing DeleteTablet for tablet ee14b0c7fa254288a9f4607883142724 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:51 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:33316
I20260619 06:27:51.861357 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 4582
I20260619 06:27:51.870713 5015 tablet_replica.cc:330] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: stopping tablet replica
I20260619 06:27:51.872262 5015 raft_consensus.cc:2229] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:51.873577 5015 raft_consensus.cc:2258] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:51.877329 5015 ts_tablet_manager.cc:1905] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:51.881594 5017 tablet_replica.cc:330] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: stopping tablet replica
I20260619 06:27:51.882006 5017 raft_consensus.cc:2229] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:51.882509 5017 raft_consensus.cc:2258] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:51.884581 5017 ts_tablet_manager.cc:1905] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:27:51 dist-test-slave-4t1f krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:27:51.887058 5015 ts_tablet_manager.cc:1918] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:51.887257 5015 log.cc:1196] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/ee14b0c7fa254288a9f4607883142724
I20260619 06:27:51.887799 5015 ts_tablet_manager.cc:1939] T ee14b0c7fa254288a9f4607883142724 P 2afb7f4029694bda9e5281f91a444827: Deleting consensus metadata
I20260619 06:27:51.889632 4506 catalog_manager.cc:4841] TS 2afb7f4029694bda9e5281f91a444827 (127.0.97.67:43877): tablet ee14b0c7fa254288a9f4607883142724 (table test-table [id=69a74e13de0d415fa4d6b449a2d7bb43]) successfully deleted
I20260619 06:27:51.892093 5017 ts_tablet_manager.cc:1918] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:51.892370 5017 log.cc:1196] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/ee14b0c7fa254288a9f4607883142724
I20260619 06:27:51.893050 5017 ts_tablet_manager.cc:1939] T ee14b0c7fa254288a9f4607883142724 P 98abf207651f4584afffdc981b5ea706: Deleting consensus metadata
I20260619 06:27:51.894405 5016 tablet_replica.cc:330] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: stopping tablet replica
I20260619 06:27:51.894852 5016 raft_consensus.cc:2229] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:51.895094 4508 catalog_manager.cc:4841] TS 98abf207651f4584afffdc981b5ea706 (127.0.97.66:45139): tablet ee14b0c7fa254288a9f4607883142724 (table test-table [id=69a74e13de0d415fa4d6b449a2d7bb43]) successfully deleted
I20260619 06:27:51.895569 5016 raft_consensus.cc:2258] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:51.897217 4987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:51.862670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.0.97.65:43071)
0619 06:27:51.862856 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:51.862865 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:51.874424 (+ 11559us) server_negotiation.cc:407] Connection header received
0619 06:27:51.874638 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:51.874646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:51.874749 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:51.874871 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:51.876658 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.877894 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.879981 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:51.880971 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:51.881113 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:51.889528 (+ 8415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:51.889545 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:51.889569 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:51.889604 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:51.893858 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:51.894887 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:51.894898 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:51.894904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:51.895106 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:51.895603 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:51.895610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:51.895613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:51.896034 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:51.896345 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:51.896618 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:27:51.896847 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 06:27:51.897815 5016 ts_tablet_manager.cc:1905] T ee14b0c7fa254288a9f4607883142724 P 9d755efcea9f40319275df2e22fda3d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:51.897920 4704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:51.991129 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 4582
W20260619 06:27:52.004283 4505 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:43071 (error 108)
I20260619 06:27:52.004606 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 4715
I20260619 06:27:52.007341 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.005778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33724)
0619 06:27:52.006919 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:52.007062 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1022,"thread_start_us":896,"threads_started":1}
W20260619 06:27:52.007536 4505 catalog_manager.cc:4584] TS 9d755efcea9f40319275df2e22fda3d8 (127.0.97.65:43071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee14b0c7fa254288a9f4607883142724: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Jun 19 06:27:52 dist-test-slave-4t1f krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:27:52.019006 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.005356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33324 (local address 127.0.97.66:45139)
0619 06:27:52.005521 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:52.005526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:52.005542 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:27:52.005725 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:52.005730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:52.005819 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:52.005941 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:52.007312 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:52.008215 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:52.009514 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:52.009885 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:52.009975 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:52.013630 (+ 3655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:52.013642 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:52.013656 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:52.013685 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:52.016403 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:52.016960 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:52.016966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:52.016968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:52.017143 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:52.017516 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:52.017521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:52.017524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:52.017877 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:52.018172 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:52.018469 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 06:27:52.018676 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:27:52.019587 4837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:52.060925 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.060498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33738)
0619 06:27:52.060711 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:52.060789 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 06:27:52.061980 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.061628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33748)
0619 06:27:52.061794 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:52.061858 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:27:52.109910 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 4715
I20260619 06:27:52.122514 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.122103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33752)
0619 06:27:52.122292 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:52.122384 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:27:52.123468 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.123157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33768)
0619 06:27:52.123282 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:52.123344 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:27:52.126375 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 4847
Jun 19 06:27:52 dist-test-slave-4t1f krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:27:52.139847 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.127006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55250 (local address 127.0.97.67:43877)
0619 06:27:52.127156 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:52.127161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:52.127177 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:27:52.127310 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:52.127314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:52.127379 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:52.127470 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:52.128647 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:52.129385 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:52.130537 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:52.130915 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:52.131001 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:52.134920 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:52.134933 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:52.134950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:27:52.134979 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:52.137696 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:52.138215 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:52.138222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:52.138224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:52.138344 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:27:52.138636 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:27:52.138639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:27:52.138640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:27:52.138962 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:52.139156 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:52.139355 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 06:27:52.139491 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:27:52.140435 4969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:52.204028 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.203519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33776)
0619 06:27:52.203799 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:52.203898 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 06:27:52.205152 5021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.204699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43071 (local address 127.0.0.1:33790)
0619 06:27:52.204908 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:52.204971 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:27:52.227212 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 4847
I20260619 06:27:52.243064 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 4490
I20260619 06:27:52.244130 4551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:52.348122 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 4490
2026-06-19T06:27:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4273 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T06:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:52Z Disabled control of system clock
I20260619 06:27:52.411993 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34917
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:34917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:27:52.717365 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:52.717626 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:52.717706 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:52.724625 5028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:52.724722 5028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:52.724761 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:52.724817 5028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:52.724867 5028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:27:52.733525 5028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:34917
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:34917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:52.734627 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:52.736310 5028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:52.747462 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:52.747545 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:52.747439 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:52.748247 5028 server_base.cc:1034] running on GCE node
I20260619 06:27:52.748925 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:52.750170 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:52.751430 5028 hybrid_clock.cc:648] HybridClock initialized: now 1781850472751392 us; error 44 us; skew 500 ppm
I20260619 06:27:52.751829 5028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:52.754212 5028 webserver.cc:460] Webserver started at http://127.0.97.126:36175/ using document root <none> and password file <none>
I20260619 06:27:52.754887 5028 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:52.755019 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:52.755357 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:52.758344 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "046f42577637477fafb4d95451ca7e78"
format_stamp: "Formatted at 2026-06-19 06:27:52 on dist-test-slave-4t1f"
server_key: "e482cb85ccaaef80bce182d6db4403ef"
server_key_iv: "5e92245805d5bdf1a0efefa0a2e0f9e7"
server_key_version: "encryptionkey@0"
I20260619 06:27:52.759171 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "046f42577637477fafb4d95451ca7e78"
format_stamp: "Formatted at 2026-06-19 06:27:52 on dist-test-slave-4t1f"
server_key: "e482cb85ccaaef80bce182d6db4403ef"
server_key_iv: "5e92245805d5bdf1a0efefa0a2e0f9e7"
server_key_version: "encryptionkey@0"
I20260619 06:27:52.764335 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 06:27:52.767721 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:52.769002 5028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:27:52.769217 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "046f42577637477fafb4d95451ca7e78"
format_stamp: "Formatted at 2026-06-19 06:27:52 on dist-test-slave-4t1f"
server_key: "e482cb85ccaaef80bce182d6db4403ef"
server_key_iv: "5e92245805d5bdf1a0efefa0a2e0f9e7"
server_key_version: "encryptionkey@0"
I20260619 06:27:52.769423 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:52.774689 5028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:52.775383 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:52.775664 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:52.792373 5028 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:34917
I20260619 06:27:52.792402 5093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:34917 every 8 connection(s)
I20260619 06:27:52.793706 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:52.796952 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5028
I20260619 06:27:52.797186 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:52.797329 5094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:52.797838 389 external_mini_cluster.cc:1428] Setting key cea8e1afe680c5aa96cba8fcf16e29c5
I20260619 06:27:52.807631 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:52.799766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.0.97.126:34917)
0619 06:27:52.800830 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:52.800845 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:52.800899 (+ 54us) server_negotiation.cc:407] Connection header received
0619 06:27:52.803619 (+ 2720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:52.803647 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:52.804191 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:52.804631 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:52.805721 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:52.805729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:52.805744 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:52.805761 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:52.805836 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:52.805860 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:52.806076 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:52.806213 (+ 137us) server_negotiation.cc:299] Negotiation successful
0619 06:27:52.806460 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":322,"threads_started":1}
I20260619 06:27:52.811322 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78: Bootstrap starting.
I20260619 06:27:52.814035 5094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:52.814970 5094 log.cc:824] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:52.817548 5094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78: No bootstrap required, opened a new log
I20260619 06:27:52.824373 5094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046f42577637477fafb4d95451ca7e78" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34917 } }
I20260619 06:27:52.824677 5094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:52.824743 5094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 046f42577637477fafb4d95451ca7e78, State: Initialized, Role: FOLLOWER
I20260619 06:27:52.825225 5094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046f42577637477fafb4d95451ca7e78" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34917 } }
I20260619 06:27:52.825418 5094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:52.825513 5094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:52.825618 5094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:52.827509 5094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046f42577637477fafb4d95451ca7e78" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34917 } }
I20260619 06:27:52.827898 5094 leader_election.cc:302] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 046f42577637477fafb4d95451ca7e78; no voters:
I20260619 06:27:52.828408 5094 leader_election.cc:288] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:52.828557 5099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:52.829238 5099 raft_consensus.cc:686] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [term 1 LEADER]: Becoming Leader. State: Replica: 046f42577637477fafb4d95451ca7e78, State: Running, Role: LEADER
I20260619 06:27:52.829797 5099 consensus_queue.cc:237] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046f42577637477fafb4d95451ca7e78" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34917 } }
I20260619 06:27:52.830539 5094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:52.832540 5101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 046f42577637477fafb4d95451ca7e78. Latest consensus state: current_term: 1 leader_uuid: "046f42577637477fafb4d95451ca7e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046f42577637477fafb4d95451ca7e78" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34917 } } }
I20260619 06:27:52.833274 5101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:52.832516 5100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "046f42577637477fafb4d95451ca7e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "046f42577637477fafb4d95451ca7e78" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 34917 } } }
I20260619 06:27:52.833806 5100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:52.834823 5106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:52.839891 5106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:52.847740 5106 catalog_manager.cc:1347] Generated new cluster ID: f44562989c034b35a03a199c94adc862
I20260619 06:27:52.847836 5106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:52.859411 5106 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:52.859932 5106 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:52.867251 5106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78: Generated new TSK 0
I20260619 06:27:52.867929 5106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:27:52.883359 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34917
--builtin_ntp_servers=127.0.97.84:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 06:27:53.197211 5118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:53.197516 5118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:53.197677 5118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:53.204753 5118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:53.204881 5118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:53.205080 5118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:27:53.214191 5118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:53.215449 5118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:53.217226 5118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:53.231166 5126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:53.231122 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:53.231204 5123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:53.231928 5118 server_base.cc:1034] running on GCE node
I20260619 06:27:53.232513 5118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:53.233520 5118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:53.234741 5118 hybrid_clock.cc:648] HybridClock initialized: now 1781850473234696 us; error 40 us; skew 500 ppm
I20260619 06:27:53.235050 5118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:53.237357 5118 webserver.cc:460] Webserver started at http://127.0.97.65:35171/ using document root <none> and password file <none>
I20260619 06:27:53.238006 5118 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:53.238142 5118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:53.238454 5118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:53.241245 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "2d5a18db8d14470f80e2fda5a8d27817"
format_stamp: "Formatted at 2026-06-19 06:27:53 on dist-test-slave-4t1f"
server_key: "f11ae74b2de22d507bc3985bb3de0ef9"
server_key_iv: "e6002198bc22cb70a6ca311bf7d42fd4"
server_key_version: "encryptionkey@0"
I20260619 06:27:53.242041 5118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "2d5a18db8d14470f80e2fda5a8d27817"
format_stamp: "Formatted at 2026-06-19 06:27:53 on dist-test-slave-4t1f"
server_key: "f11ae74b2de22d507bc3985bb3de0ef9"
server_key_iv: "e6002198bc22cb70a6ca311bf7d42fd4"
server_key_version: "encryptionkey@0"
I20260619 06:27:53.246958 5118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:27:53.250211 5132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:53.251511 5118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:53.251677 5118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "2d5a18db8d14470f80e2fda5a8d27817"
format_stamp: "Formatted at 2026-06-19 06:27:53 on dist-test-slave-4t1f"
server_key: "f11ae74b2de22d507bc3985bb3de0ef9"
server_key_iv: "e6002198bc22cb70a6ca311bf7d42fd4"
server_key_version: "encryptionkey@0"
I20260619 06:27:53.251899 5118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:53.257198 5118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:53.257915 5118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:53.258177 5118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:53.259006 5118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:53.260270 5118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:53.260396 5118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:53.260514 5118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:53.260607 5118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:53.292433 5118 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:37611
I20260619 06:27:53.292479 5244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:37611 every 8 connection(s)
I20260619 06:27:53.293768 5118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:27:53.301234 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:53.296186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:38091 (local address 127.0.97.126:34917)
0619 06:27:53.296391 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:53.296398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:53.297801 (+ 1403us) server_negotiation.cc:407] Connection header received
0619 06:27:53.299358 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:53.299363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:53.299411 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:53.299498 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:53.300640 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:53.300643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:53.300646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:53.300661 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:53.300689 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:53.300700 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:53.300787 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:53.300939 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 06:27:53.300982 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:27:53.302125 5247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:53.296752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34917 (local address 127.0.97.65:38091)
0619 06:27:53.297648 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:53.297682 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:53.299131 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:53.299700 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:53.299708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:53.299793 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:53.300376 (+ 583us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:53.300378 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:53.300412 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:53.300488 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:53.300778 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:53.300785 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:53.300790 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:27:53.300981 (+ 191us) client_negotiation.cc:240] Negotiation successful
0619 06:27:53.301228 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":304,"threads_started":1}
I20260619 06:27:53.302368 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5118
I20260619 06:27:53.302595 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:27:53.303198 389 external_mini_cluster.cc:1428] Setting key db30cd6107c8077a51e9b27199f424d3
I20260619 06:27:53.303638 5245 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34917
I20260619 06:27:53.303908 5245 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:53.304437 5245 heartbeater.cc:511] Master 127.0.97.126:34917 requested a full tablet report, sending...
I20260619 06:27:53.305816 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34917
--builtin_ntp_servers=127.0.97.84:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:27:53.306582 5059 ts_manager.cc:194] Registered new tserver with Master: 2d5a18db8d14470f80e2fda5a8d27817 (127.0.97.65:37611)
W20260619 06:27:53.616353 5248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:53.616611 5248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:53.616707 5248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:53.624969 5248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:53.625099 5248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:53.625238 5248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:27:53.634248 5248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:53.635463 5248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:53.637215 5248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:53.649487 5256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:53.649521 5253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:53.649555 5254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:53.650228 5248 server_base.cc:1034] running on GCE node
I20260619 06:27:53.650771 5248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:53.651562 5248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:53.652781 5248 hybrid_clock.cc:648] HybridClock initialized: now 1781850473652748 us; error 45 us; skew 500 ppm
I20260619 06:27:53.653110 5248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:53.655290 5248 webserver.cc:460] Webserver started at http://127.0.97.66:46197/ using document root <none> and password file <none>
I20260619 06:27:53.655977 5248 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:53.656095 5248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:53.656396 5248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:53.659072 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "2bef418001874a1188884093516a849a"
format_stamp: "Formatted at 2026-06-19 06:27:53 on dist-test-slave-4t1f"
server_key: "382251b54ffdb44c66ce7fd58ea0f397"
server_key_iv: "020ff5c3fdcb0725eedf021d7fc9ce28"
server_key_version: "encryptionkey@0"
I20260619 06:27:53.659818 5248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "2bef418001874a1188884093516a849a"
format_stamp: "Formatted at 2026-06-19 06:27:53 on dist-test-slave-4t1f"
server_key: "382251b54ffdb44c66ce7fd58ea0f397"
server_key_iv: "020ff5c3fdcb0725eedf021d7fc9ce28"
server_key_version: "encryptionkey@0"
I20260619 06:27:53.664405 5248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 06:27:53.667512 5262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:53.668712 5248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:27:53.668875 5248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "2bef418001874a1188884093516a849a"
format_stamp: "Formatted at 2026-06-19 06:27:53 on dist-test-slave-4t1f"
server_key: "382251b54ffdb44c66ce7fd58ea0f397"
server_key_iv: "020ff5c3fdcb0725eedf021d7fc9ce28"
server_key_version: "encryptionkey@0"
I20260619 06:27:53.669121 5248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:53.674474 5248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:53.675197 5248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:53.675453 5248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:53.676296 5248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:53.677588 5248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:53.677716 5248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:53.677846 5248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:53.677932 5248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:53.709165 5248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:40003
I20260619 06:27:53.709252 5374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:40003 every 8 connection(s)
I20260619 06:27:53.710428 5248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:27:53.713469 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5248
I20260619 06:27:53.713646 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:27:53.714067 389 external_mini_cluster.cc:1428] Setting key 12087b9f65d79e664ce455ffa48ad9bd
I20260619 06:27:53.716704 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34917
--builtin_ntp_servers=127.0.97.84:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 06:27:53.718976 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:53.712806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:38727 (local address 127.0.97.126:34917)
0619 06:27:53.712961 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:53.712965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:53.714506 (+ 1541us) server_negotiation.cc:407] Connection header received
0619 06:27:53.716324 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:53.716332 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:53.716396 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:53.716506 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:53.717931 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:53.717965 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:53.717974 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:53.717997 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:53.718055 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:53.718074 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:53.718195 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:53.718468 (+ 273us) server_negotiation.cc:299] Negotiation successful
0619 06:27:53.718556 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:27:53.719655 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:53.713351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34917 (local address 127.0.97.66:38727)
0619 06:27:53.714341 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:53.714389 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:53.716054 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:53.716698 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:53.716705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:53.716789 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:53.717544 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:53.717547 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:53.717582 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:53.717668 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:53.718193 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:53.718207 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:53.718217 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:27:53.718540 (+ 323us) client_negotiation.cc:240] Negotiation successful
0619 06:27:53.718797 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":317,"threads_started":1}
I20260619 06:27:53.720953 5375 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34917
I20260619 06:27:53.721243 5375 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:53.721760 5375 heartbeater.cc:511] Master 127.0.97.126:34917 requested a full tablet report, sending...
I20260619 06:27:53.723110 5059 ts_manager.cc:194] Registered new tserver with Master: 2bef418001874a1188884093516a849a (127.0.97.66:40003)
W20260619 06:27:54.025951 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:54.026193 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:54.026365 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:54.033524 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:54.033608 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:54.033779 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:27:54.042615 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:34917
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:54.043782 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:54.045456 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:54.057788 5383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:54.057756 5386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:54.057765 5384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:54.058677 5378 server_base.cc:1034] running on GCE node
I20260619 06:27:54.059194 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:54.060011 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:54.061205 5378 hybrid_clock.cc:648] HybridClock initialized: now 1781850474061168 us; error 46 us; skew 500 ppm
I20260619 06:27:54.061512 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:54.063488 5378 webserver.cc:460] Webserver started at http://127.0.97.67:34829/ using document root <none> and password file <none>
I20260619 06:27:54.064090 5378 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:54.064222 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:54.064522 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:54.067152 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "f58afe1de362448e80997763c6b927fe"
format_stamp: "Formatted at 2026-06-19 06:27:54 on dist-test-slave-4t1f"
server_key: "6d508e72c3612a34f9a80d72f86174c0"
server_key_iv: "aacacf27b0dcc2f12f28f73b2928b982"
server_key_version: "encryptionkey@0"
I20260619 06:27:54.067942 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "f58afe1de362448e80997763c6b927fe"
format_stamp: "Formatted at 2026-06-19 06:27:54 on dist-test-slave-4t1f"
server_key: "6d508e72c3612a34f9a80d72f86174c0"
server_key_iv: "aacacf27b0dcc2f12f28f73b2928b982"
server_key_version: "encryptionkey@0"
I20260619 06:27:54.072650 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:27:54.076378 5392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:54.078088 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:54.078249 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "f58afe1de362448e80997763c6b927fe"
format_stamp: "Formatted at 2026-06-19 06:27:54 on dist-test-slave-4t1f"
server_key: "6d508e72c3612a34f9a80d72f86174c0"
server_key_iv: "aacacf27b0dcc2f12f28f73b2928b982"
server_key_version: "encryptionkey@0"
I20260619 06:27:54.078502 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:54.085399 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:54.086107 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:54.086377 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:54.087180 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:27:54.088474 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:27:54.088590 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:54.088711 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:27:54.088804 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:54.119822 5378 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40109
I20260619 06:27:54.119876 5504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40109 every 8 connection(s)
I20260619 06:27:54.121206 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:27:54.125479 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5378
I20260619 06:27:54.125649 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:27:54.126066 389 external_mini_cluster.cc:1428] Setting key 477aa458e94b001ed3822758d24b5eea
I20260619 06:27:54.128886 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.123536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:58107 (local address 127.0.97.126:34917)
0619 06:27:54.123726 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.123730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.125033 (+ 1303us) server_negotiation.cc:407] Connection header received
0619 06:27:54.126853 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.126857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.126903 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.126981 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.128201 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.128208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.128213 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.128229 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.128271 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.128287 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.128405 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.128588 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.128649 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:27:54.129788 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.124054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:34917 (local address 127.0.97.67:58107)
0619 06:27:54.124884 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:54.124917 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:54.126560 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:54.127210 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:54.127218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:54.127305 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:54.127940 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:54.127942 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:54.127976 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:54.128055 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:54.128381 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:54.128387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:54.128393 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:27:54.128664 (+ 271us) client_negotiation.cc:240] Negotiation successful
0619 06:27:54.128907 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":235,"threads_started":1}
I20260619 06:27:54.130996 5505 heartbeater.cc:347] Connected to a master server at 127.0.97.126:34917
I20260619 06:27:54.131265 5505 heartbeater.cc:464] Registering TS with master...
I20260619 06:27:54.131765 5505 heartbeater.cc:511] Master 127.0.97.126:34917 requested a full tablet report, sending...
I20260619 06:27:54.132993 5059 ts_manager.cc:194] Registered new tserver with Master: f58afe1de362448e80997763c6b927fe (127.0.97.67:40109)
I20260619 06:27:54.140594 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:27:54.147033 5097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.144894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.0.97.126:34917)
0619 06:27:54.145097 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.145105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.145527 (+ 422us) server_negotiation.cc:407] Connection header received
0619 06:27:54.145635 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.145638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.145681 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.145764 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.146593 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.146597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.146599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.146610 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.146632 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.146643 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.146724 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.146823 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.146861 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:27:54.150444 5059 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:27:54.157002 5059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:27:54.174980 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.168099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:40003 (local address 127.0.0.1:42368)
0619 06:27:54.171240 (+ 3141us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:54.171256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:54.171347 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:54.172185 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:54.172189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:54.172260 (+ 71us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:54.173336 (+ 1076us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:54.173341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:54.173368 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:54.173456 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:54.174229 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:54.174239 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:54.174247 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 06:27:54.174406 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 06:27:54.174494 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3007,"thread_start_us":462,"threads_started":1}
I20260619 06:27:54.175813 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.168747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.0.97.66:40003)
0619 06:27:54.170490 (+ 1743us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.170502 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.171454 (+ 952us) server_negotiation.cc:407] Connection header received
0619 06:27:54.171522 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.171528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.171712 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.171857 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.173913 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.173924 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.173940 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.173960 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.174018 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.174041 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.174176 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.175381 (+ 1205us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.175477 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1528,"thread_start_us":1367,"threads_started":1}
I20260619 06:27:54.176450 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.167876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.0.97.65:37611)
0619 06:27:54.170902 (+ 3026us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.170920 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.170953 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:27:54.171135 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.171145 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.171385 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.171522 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.174763 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.174769 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.174780 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.174821 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.174875 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.174892 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.175011 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.175746 (+ 735us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.175834 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2855,"thread_start_us":2750,"threads_started":1}
I20260619 06:27:54.175624 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.167710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:37611 (local address 127.0.0.1:54368)
0619 06:27:54.170765 (+ 3055us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:54.170811 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:54.170985 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:54.171985 (+ 1000us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:54.171992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:54.173424 (+ 1432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:54.174557 (+ 1133us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:54.174559 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:54.174572 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:54.174609 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:54.175062 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:54.175069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:54.175076 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:27:54.175202 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 06:27:54.175255 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2776,"thread_start_us":3215,"threads_started":1}
I20260619 06:27:54.179979 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.171294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40109 (local address 127.0.0.1:51860)
0619 06:27:54.176899 (+ 5605us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:54.176918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:54.176986 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:54.178416 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:54.178421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:54.178451 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:54.178971 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:54.178974 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:54.178990 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:54.179033 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:54.179591 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:54.179596 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:54.179600 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 06:27:54.179695 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:27:54.179740 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5516,"thread_start_us":6170,"threads_started":1}
I20260619 06:27:54.180123 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.171545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.97.67:40109)
0619 06:27:54.177974 (+ 6429us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.177982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.178010 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:27:54.178071 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.178077 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.178255 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.178398 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.179364 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.179371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.179382 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.179400 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.179450 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.179474 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.179645 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.179703 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.179814 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6277,"thread_start_us":6510,"threads_started":1}
I20260619 06:27:54.181342 5180 tablet_service.cc:1467] Processing CreateTablet for tablet ba1722fcfa574075b054137323ef9973 (DEFAULT_TABLE table=test-table [id=0795db4ccfc34a3092fc88acd49c89ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:54.182542 5180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1722fcfa574075b054137323ef9973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:54.183171 5440 tablet_service.cc:1467] Processing CreateTablet for tablet ba1722fcfa574075b054137323ef9973 (DEFAULT_TABLE table=test-table [id=0795db4ccfc34a3092fc88acd49c89ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:54.184298 5440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1722fcfa574075b054137323ef9973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:54.180889 5310 tablet_service.cc:1467] Processing CreateTablet for tablet ba1722fcfa574075b054137323ef9973 (DEFAULT_TABLE table=test-table [id=0795db4ccfc34a3092fc88acd49c89ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:27:54.192749 5310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba1722fcfa574075b054137323ef9973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:54.194113 5523 tablet_bootstrap.cc:492] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Bootstrap starting.
I20260619 06:27:54.197218 5523 tablet_bootstrap.cc:654] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:54.197791 5524 tablet_bootstrap.cc:492] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Bootstrap starting.
I20260619 06:27:54.198845 5523 log.cc:824] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:54.201071 5524 tablet_bootstrap.cc:654] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:54.202054 5523 tablet_bootstrap.cc:492] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: No bootstrap required, opened a new log
I20260619 06:27:54.202311 5523 ts_tablet_manager.cc:1397] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 06:27:54.202298 5524 log.cc:824] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:54.205143 5524 tablet_bootstrap.cc:492] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: No bootstrap required, opened a new log
I20260619 06:27:54.205411 5524 ts_tablet_manager.cc:1397] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 06:27:54.205889 5527 tablet_bootstrap.cc:492] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Bootstrap starting.
I20260619 06:27:54.209100 5527 tablet_bootstrap.cc:654] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:54.209998 5527 log.cc:824] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:54.212219 5527 tablet_bootstrap.cc:492] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: No bootstrap required, opened a new log
I20260619 06:27:54.212448 5527 ts_tablet_manager.cc:1397] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:27:54.212481 5523 raft_consensus.cc:348] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.212899 5523 raft_consensus.cc:374] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:54.212988 5523 raft_consensus.cc:729] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5a18db8d14470f80e2fda5a8d27817, State: Initialized, Role: FOLLOWER
I20260619 06:27:54.213588 5523 consensus_queue.cc:260] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.214627 5245 heartbeater.cc:503] Master 127.0.97.126:34917 was elected leader, sending a full tablet report...
I20260619 06:27:54.215193 5523 ts_tablet_manager.cc:1428] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260619 06:27:54.215555 5524 raft_consensus.cc:348] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.215981 5524 raft_consensus.cc:374] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:54.216086 5524 raft_consensus.cc:729] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f58afe1de362448e80997763c6b927fe, State: Initialized, Role: FOLLOWER
I20260619 06:27:54.216655 5524 consensus_queue.cc:260] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.218523 5505 heartbeater.cc:503] Master 127.0.97.126:34917 was elected leader, sending a full tablet report...
I20260619 06:27:54.219206 5524 ts_tablet_manager.cc:1428] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260619 06:27:54.222842 5527 raft_consensus.cc:348] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.223112 5527 raft_consensus.cc:374] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:54.223168 5527 raft_consensus.cc:729] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bef418001874a1188884093516a849a, State: Initialized, Role: FOLLOWER
I20260619 06:27:54.223553 5527 consensus_queue.cc:260] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.224335 5375 heartbeater.cc:503] Master 127.0.97.126:34917 was elected leader, sending a full tablet report...
I20260619 06:27:54.224862 5527 ts_tablet_manager.cc:1428] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260619 06:27:54.295964 5246 tablet.cc:2369] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:27:54.372826 5506 tablet.cc:2369] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:27:54.462404 5376 tablet.cc:2369] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:27:54.505684 5529 raft_consensus.cc:482] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:27:54.506032 5529 raft_consensus.cc:504] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.507997 5529 leader_election.cc:288] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2bef418001874a1188884093516a849a (127.0.97.66:40003), f58afe1de362448e80997763c6b927fe (127.0.97.67:40109)
I20260619 06:27:54.511811 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.508307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:44435 (local address 127.0.97.66:40003)
0619 06:27:54.508512 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.508521 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.509188 (+ 667us) server_negotiation.cc:407] Connection header received
0619 06:27:54.509937 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.509945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.510004 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.510099 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.511171 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.511176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.511179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.511196 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.511231 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.511245 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.511375 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.511493 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.511550 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:27:54.511799 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.508650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56213 (local address 127.0.97.67:40109)
0619 06:27:54.508938 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.508945 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.509699 (+ 754us) server_negotiation.cc:407] Connection header received
0619 06:27:54.509818 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.509822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.509900 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.510010 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.511168 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.511174 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.511178 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.511196 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.511231 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.511245 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.511378 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.511493 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.511550 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 06:27:54.512168 5533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.508303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:40109 (local address 127.0.97.65:56213)
0619 06:27:54.509611 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:54.509625 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:54.509719 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:54.510039 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:54.510042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:54.510082 (+ 40us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:54.510956 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:54.510958 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:54.510968 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:54.511000 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:54.511363 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:54.511367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:54.511372 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:27:54.511522 (+ 150us) client_negotiation.cc:240] Negotiation successful
0619 06:27:54.511592 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1194,"thread_start_us":1018,"threads_started":1}
I20260619 06:27:54.512189 5532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.508189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:40003 (local address 127.0.97.65:44435)
0619 06:27:54.509051 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:54.509096 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:54.509822 (+ 726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:54.510136 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:54.510142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:54.510167 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:27:54.510930 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:27:54.510935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:27:54.510954 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:27:54.511000 (+ 46us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:27:54.511363 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:27:54.511367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:27:54.511372 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:27:54.511546 (+ 174us) client_negotiation.cc:240] Negotiation successful
0619 06:27:54.511603 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"mutex_wait_us":17,"thread_start_us":515,"threads_started":1}
I20260619 06:27:54.513286 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1722fcfa574075b054137323ef9973" candidate_uuid: "2d5a18db8d14470f80e2fda5a8d27817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bef418001874a1188884093516a849a" is_pre_election: true
I20260619 06:27:54.513265 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1722fcfa574075b054137323ef9973" candidate_uuid: "2d5a18db8d14470f80e2fda5a8d27817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58afe1de362448e80997763c6b927fe" is_pre_election: true
I20260619 06:27:54.513779 5330 raft_consensus.cc:2454] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d5a18db8d14470f80e2fda5a8d27817 in term 0.
I20260619 06:27:54.513790 5460 raft_consensus.cc:2454] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d5a18db8d14470f80e2fda5a8d27817 in term 0.
I20260619 06:27:54.514420 5135 leader_election.cc:302] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d5a18db8d14470f80e2fda5a8d27817, f58afe1de362448e80997763c6b927fe; no voters:
I20260619 06:27:54.514868 5529 raft_consensus.cc:2781] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:27:54.515007 5529 raft_consensus.cc:482] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:27:54.515122 5529 raft_consensus.cc:3037] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:54.517266 5529 raft_consensus.cc:504] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.517858 5529 leader_election.cc:288] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [CANDIDATE]: Term 1 election: Requested vote from peers 2bef418001874a1188884093516a849a (127.0.97.66:40003), f58afe1de362448e80997763c6b927fe (127.0.97.67:40109)
I20260619 06:27:54.518271 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1722fcfa574075b054137323ef9973" candidate_uuid: "2d5a18db8d14470f80e2fda5a8d27817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2bef418001874a1188884093516a849a"
I20260619 06:27:54.518294 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1722fcfa574075b054137323ef9973" candidate_uuid: "2d5a18db8d14470f80e2fda5a8d27817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f58afe1de362448e80997763c6b927fe"
I20260619 06:27:54.518471 5330 raft_consensus.cc:3037] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:54.518476 5460 raft_consensus.cc:3037] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:54.520501 5330 raft_consensus.cc:2454] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d5a18db8d14470f80e2fda5a8d27817 in term 1.
I20260619 06:27:54.520516 5460 raft_consensus.cc:2454] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d5a18db8d14470f80e2fda5a8d27817 in term 1.
I20260619 06:27:54.520924 5133 leader_election.cc:302] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2bef418001874a1188884093516a849a, 2d5a18db8d14470f80e2fda5a8d27817, f58afe1de362448e80997763c6b927fe; no voters:
I20260619 06:27:54.521238 5529 raft_consensus.cc:2781] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:54.521584 5529 raft_consensus.cc:686] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 1 LEADER]: Becoming Leader. State: Replica: 2d5a18db8d14470f80e2fda5a8d27817, State: Running, Role: LEADER
I20260619 06:27:54.522025 5529 consensus_queue.cc:237] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } }
I20260619 06:27:54.526435 5059 catalog_manager.cc:5495] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d5a18db8d14470f80e2fda5a8d27817 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "2d5a18db8d14470f80e2fda5a8d27817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d5a18db8d14470f80e2fda5a8d27817" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 37611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 } health_report { overall_health: UNKNOWN } } }
I20260619 06:27:54.538098 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.536000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.0.97.65:37611)
0619 06:27:54.536146 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.536150 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.536277 (+ 127us) server_negotiation.cc:407] Connection header received
0619 06:27:54.536498 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.536505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.536586 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.536667 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.537672 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.537675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.537678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.537690 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.537715 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.537725 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.537801 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.537872 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.537911 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:27:54.545794 5330 raft_consensus.cc:1261] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 1 FOLLOWER]: Refusing update from remote peer 2d5a18db8d14470f80e2fda5a8d27817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:54.545794 5460 raft_consensus.cc:1261] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 1 FOLLOWER]: Refusing update from remote peer 2d5a18db8d14470f80e2fda5a8d27817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:27:54.546765 5534 consensus_queue.cc:1035] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f58afe1de362448e80997763c6b927fe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 40109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:54.547130 5529 consensus_queue.cc:1035] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2bef418001874a1188884093516a849a" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 40003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:27:54.558612 5537 mvcc.cc:205] Tried to move back new op lower bound from 7298459543730184192 to 7298459543645454336. Current Snapshot: MvccSnapshot[applied={T|T < 7298459543730184192}]
I20260619 06:27:54.559420 5538 mvcc.cc:205] Tried to move back new op lower bound from 7298459543730184192 to 7298459543645454336. Current Snapshot: MvccSnapshot[applied={T|T < 7298459543730184192}]
I20260619 06:27:54.571034 5059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:54.571288 5059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:27:54.574976 5059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 046f42577637477fafb4d95451ca7e78: Sending DeleteTablet for 3 replicas of tablet ba1722fcfa574075b054137323ef9973
I20260619 06:27:54.575934 5180 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1722fcfa574075b054137323ef9973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:54 UTC) from {username='slave'} at 127.0.0.1:54368
I20260619 06:27:54.576113 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1722fcfa574075b054137323ef9973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:54 UTC) from {username='slave'} at 127.0.0.1:51860
I20260619 06:27:54.576452 5310 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1722fcfa574075b054137323ef9973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:27:54 UTC) from {username='slave'} at 127.0.0.1:42368
I20260619 06:27:54.578423 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5118
I20260619 06:27:54.587242 5549 tablet_replica.cc:330] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: stopping tablet replica
I20260619 06:27:54.587435 5547 tablet_replica.cc:330] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: stopping tablet replica
I20260619 06:27:54.587822 5549 raft_consensus.cc:2229] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:54.587846 5547 raft_consensus.cc:2229] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:27:54.588362 5549 raft_consensus.cc:2258] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:54.588467 5547 raft_consensus.cc:2258] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:54.589087 5548 tablet_replica.cc:330] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: stopping tablet replica
I20260619 06:27:54.589537 5548 raft_consensus.cc:2229] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:27:54.590009 5548 raft_consensus.cc:2258] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:27:54.590510 5547 ts_tablet_manager.cc:1905] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:54.590574 5549 ts_tablet_manager.cc:1905] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:54.591979 5548 ts_tablet_manager.cc:1905] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:27:54.597047 5547 ts_tablet_manager.cc:1918] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:54.597182 5549 ts_tablet_manager.cc:1918] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:54.597263 5547 log.cc:1196] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/ba1722fcfa574075b054137323ef9973
I20260619 06:27:54.597390 5549 log.cc:1196] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/ba1722fcfa574075b054137323ef9973
I20260619 06:27:54.597836 5547 ts_tablet_manager.cc:1939] T ba1722fcfa574075b054137323ef9973 P 2d5a18db8d14470f80e2fda5a8d27817: Deleting consensus metadata
I20260619 06:27:54.597951 5549 ts_tablet_manager.cc:1939] T ba1722fcfa574075b054137323ef9973 P 2bef418001874a1188884093516a849a: Deleting consensus metadata
I20260619 06:27:54.598423 5548 ts_tablet_manager.cc:1918] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:27:54.599586 5045 catalog_manager.cc:4841] TS 2d5a18db8d14470f80e2fda5a8d27817 (127.0.97.65:37611): tablet ba1722fcfa574075b054137323ef9973 (table test-table [id=0795db4ccfc34a3092fc88acd49c89ed]) successfully deleted
I20260619 06:27:54.599900 5043 catalog_manager.cc:4841] TS 2bef418001874a1188884093516a849a (127.0.97.66:40003): tablet ba1722fcfa574075b054137323ef9973 (table test-table [id=0795db4ccfc34a3092fc88acd49c89ed]) successfully deleted
I20260619 06:27:54.600100 5518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.579675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54380 (local address 127.0.97.65:37611)
0619 06:27:54.579886 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.579896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.597476 (+ 17580us) server_negotiation.cc:407] Connection header received
0619 06:27:54.597947 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.597952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.598013 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.598092 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.599310 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.599328 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.599336 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.599357 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.599395 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.599413 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.599509 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.599751 (+ 242us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.599810 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:27:54.600747 5548 log.cc:1196] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/ba1722fcfa574075b054137323ef9973
I20260619 06:27:54.600829 5240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:54.601271 5548 ts_tablet_manager.cc:1939] T ba1722fcfa574075b054137323ef9973 P f58afe1de362448e80997763c6b927fe: Deleting consensus metadata
I20260619 06:27:54.602790 5045 catalog_manager.cc:4841] TS f58afe1de362448e80997763c6b927fe (127.0.97.67:40109): tablet ba1722fcfa574075b054137323ef9973 (table test-table [id=0795db4ccfc34a3092fc88acd49c89ed]) successfully deleted
I20260619 06:27:54.695256 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5118
I20260619 06:27:54.712962 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5248
I20260619 06:27:54.715785 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.713726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42370 (local address 127.0.97.66:40003)
0619 06:27:54.713898 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.713903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.713959 (+ 56us) server_negotiation.cc:407] Connection header received
0619 06:27:54.714098 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.714103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.714200 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.714314 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.715179 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.715185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.715190 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.715206 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.715240 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.715253 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.715383 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.715435 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.715485 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:27:54.716404 5370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:54.801717 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5248
I20260619 06:27:54.819084 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5378
I20260619 06:27:54.821573 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:54.819794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.0.97.67:40109)
0619 06:27:54.819958 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:54.819962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:54.819979 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:27:54.820086 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:54.820090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:54.820174 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:54.820257 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:54.821105 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:54.821110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:54.821115 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:54.821138 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:54.821164 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:54.821175 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:54.821268 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:54.821307 (+ 39us) server_negotiation.cc:299] Negotiation successful
0619 06:27:54.821347 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:27:54.822144 5500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:54.906734 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5378
I20260619 06:27:54.922328 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5028
I20260619 06:27:54.923431 5089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:55.026384 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5028
2026-06-19T06:27:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2676 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 06:27:55.065802 389 mini_oidc.cc:152] Starting JWKS server
I20260619 06:27:55.068228 389 webserver.cc:460] Webserver started at https://127.0.0.1:44197/ using document root <none> and password file <none>
I20260619 06:27:55.068512 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:27:55.069166 389 webserver.cc:460] Webserver started at http://127.0.0.1:41939/ using document root <none> and password file <none>
2026-06-19T06:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:55Z Disabled control of system clock
I20260619 06:27:55.093521 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43837
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43837
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44197/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/testchainca.cert with env {}
W20260619 06:27:55.401043 5560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:55.401360 5560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:55.401453 5560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:55.408926 5560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:55.409025 5560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:55.409106 5560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:55.409180 5560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:55.409225 5560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:27:55.409291 5560 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:27:55.409334 5560 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44197/jwks.json
I20260619 06:27:55.418883 5560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43837
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44197/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:55.419989 5560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:55.421226 5560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:55.432446 5565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:55.432500 5566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:55.432528 5568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:55.433139 5560 server_base.cc:1034] running on GCE node
I20260619 06:27:55.434006 5560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:55.435240 5560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:55.436465 5560 hybrid_clock.cc:648] HybridClock initialized: now 1781850475436428 us; error 54 us; skew 500 ppm
I20260619 06:27:55.436777 5560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:55.438913 5560 webserver.cc:460] Webserver started at http://127.0.97.126:44983/ using document root <none> and password file <none>
I20260619 06:27:55.439548 5560 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:55.439674 5560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:55.439968 5560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:55.442638 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "8fccf3eddef64423bc3ada7500625215"
format_stamp: "Formatted at 2026-06-19 06:27:55 on dist-test-slave-4t1f"
server_key: "d49b6494e481d00f555e10e1ef7ec0d4"
server_key_iv: "0ff67b403a408d85d2ce6e4202fb757d"
server_key_version: "encryptionkey@0"
I20260619 06:27:55.443394 5560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "8fccf3eddef64423bc3ada7500625215"
format_stamp: "Formatted at 2026-06-19 06:27:55 on dist-test-slave-4t1f"
server_key: "d49b6494e481d00f555e10e1ef7ec0d4"
server_key_iv: "0ff67b403a408d85d2ce6e4202fb757d"
server_key_version: "encryptionkey@0"
I20260619 06:27:55.448167 5560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:27:55.451308 5574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:55.452462 5560 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:55.452638 5560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "8fccf3eddef64423bc3ada7500625215"
format_stamp: "Formatted at 2026-06-19 06:27:55 on dist-test-slave-4t1f"
server_key: "d49b6494e481d00f555e10e1ef7ec0d4"
server_key_iv: "0ff67b403a408d85d2ce6e4202fb757d"
server_key_version: "encryptionkey@0"
I20260619 06:27:55.452836 5560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:55.475481 5560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:55.476250 5560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:55.476531 5560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:55.494582 5560 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43837
I20260619 06:27:55.494601 5625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43837 every 8 connection(s)
I20260619 06:27:55.495976 5560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:55.498893 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5560
I20260619 06:27:55.499109 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:55.499665 5626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:55.499782 389 external_mini_cluster.cc:1428] Setting key feb14ebeceabfa257f743acbc554eafe
I20260619 06:27:55.514528 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:55.501644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.0.97.126:43837)
0619 06:27:55.502694 (+ 1050us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:55.502714 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:55.502760 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:27:55.504552 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:55.504578 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:55.505167 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:55.505595 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:55.506086 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:55.508069 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:55.509456 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:55.510468 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:55.510668 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:55.511847 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:55.511858 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:55.511879 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:55.512591 (+ 712us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:55.512679 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:55.512702 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:55.512806 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:55.512990 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 06:27:55.513313 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":313,"threads_started":1}
I20260619 06:27:55.514545 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215: Bootstrap starting.
I20260619 06:27:55.517650 5626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:55.518090 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:27:55.518687 5626 log.cc:824] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:55.521349 5626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215: No bootstrap required, opened a new log
I20260619 06:27:55.528517 5626 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fccf3eddef64423bc3ada7500625215" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43837 } }
I20260619 06:27:55.528833 5626 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:55.528949 5626 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fccf3eddef64423bc3ada7500625215, State: Initialized, Role: FOLLOWER
I20260619 06:27:55.529464 5626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fccf3eddef64423bc3ada7500625215" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43837 } }
I20260619 06:27:55.529677 5626 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:55.529821 5626 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:55.529980 5626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:55.531967 5626 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fccf3eddef64423bc3ada7500625215" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43837 } }
I20260619 06:27:55.532490 5626 leader_election.cc:302] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fccf3eddef64423bc3ada7500625215; no voters:
I20260619 06:27:55.533025 5626 leader_election.cc:288] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:55.533234 5631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:55.534561 5631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [term 1 LEADER]: Becoming Leader. State: Replica: 8fccf3eddef64423bc3ada7500625215, State: Running, Role: LEADER
I20260619 06:27:55.535059 5631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fccf3eddef64423bc3ada7500625215" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43837 } }
I20260619 06:27:55.535306 5626 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:55.537628 5633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fccf3eddef64423bc3ada7500625215. Latest consensus state: current_term: 1 leader_uuid: "8fccf3eddef64423bc3ada7500625215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fccf3eddef64423bc3ada7500625215" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43837 } } }
I20260619 06:27:55.537987 5633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:55.538451 5632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fccf3eddef64423bc3ada7500625215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fccf3eddef64423bc3ada7500625215" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43837 } } }
I20260619 06:27:55.538807 5632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:55.539114 5638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:55.547389 5638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:55.556766 5638 catalog_manager.cc:1347] Generated new cluster ID: 1a7d165623954bce9d13121e057b59e4
I20260619 06:27:55.556905 5638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:55.576145 5638 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:55.577580 5638 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:55.590968 5638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fccf3eddef64423bc3ada7500625215: Generated new TSK 0
I20260619 06:27:55.591813 5638 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:27:55.629640 5657 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 06:27:55.629863 5657 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:27:55.630867 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:55.609322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37898 (local address 127.0.97.126:43837)
0619 06:27:55.609474 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:55.609478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:55.610034 (+ 556us) server_negotiation.cc:407] Connection header received
0619 06:27:55.610349 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:55.610353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:55.610414 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:55.610555 (+ 141us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:27:55.611231 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:55.611951 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:55.613401 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:55.613877 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:55.614006 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:55.614545 (+ 539us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:27:55.630203 (+ 15658us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:27:55.630563 (+ 360us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37898: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 06:27:55.631220 5629 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37898: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 06:27:55.632580 5656 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:27:55.609355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43837 (local address 127.0.0.1:37898)
0619 06:27:55.609920 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:55.609935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:55.610128 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:55.610586 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:55.610593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:55.610605 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:27:55.610992 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:55.611006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:55.612093 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:55.612100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:55.613262 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:55.613274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:55.614062 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:55.614071 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:55.614214 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:55.614266 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:27:55.630557 (+ 16291us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 06:27:55.631063 (+ 506us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:43837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":302,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.126:43837: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44197/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 06:27:55.634333 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5560
I20260619 06:27:55.635496 5621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:55.728821 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5560
2026-06-19T06:27:55Z chronyd exiting
I20260619 06:27:55.778390 389 test_util.cc:175] -----------------------------------------------
I20260619 06:27:55.778515 389 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781850433974487-389-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (716 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 06:27:55.782063 389 mini_oidc.cc:152] Starting JWKS server
I20260619 06:27:55.783839 389 webserver.cc:460] Webserver started at https://127.0.0.1:37255/ using document root <none> and password file <none>
I20260619 06:27:55.784083 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:27:55.784709 389 webserver.cc:460] Webserver started at http://127.0.0.1:39113/ using document root <none> and password file <none>
2026-06-19T06:27:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:55Z Disabled control of system clock
I20260619 06:27:55.808598 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42203
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:34525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37255/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/testchainca.cert with env {}
W20260619 06:27:56.115453 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:56.115720 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:56.115805 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:56.122844 5664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:56.122938 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:56.122979 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:56.123032 5664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:56.123075 5664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:27:56.123131 5664 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:27:56.123169 5664 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37255/jwks.json
I20260619 06:27:56.132392 5664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37255/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:56.133646 5664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:56.134801 5664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:56.145741 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:56.145783 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:56.145800 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:56.146411 5664 server_base.cc:1034] running on GCE node
I20260619 06:27:56.147054 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:56.148223 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:56.149430 5664 hybrid_clock.cc:648] HybridClock initialized: now 1781850476149394 us; error 47 us; skew 500 ppm
I20260619 06:27:56.149760 5664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:56.151873 5664 webserver.cc:460] Webserver started at http://127.0.97.126:33915/ using document root <none> and password file <none>
I20260619 06:27:56.152521 5664 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:56.152657 5664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:56.152947 5664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:56.155579 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "85ac1427853048a7949464d910698593"
format_stamp: "Formatted at 2026-06-19 06:27:56 on dist-test-slave-4t1f"
server_key: "1835aa7914d7f432433e29d273a5ef82"
server_key_iv: "f25a09ee9fb13964d6fb59ea0b602f1f"
server_key_version: "encryptionkey@0"
I20260619 06:27:56.156312 5664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "85ac1427853048a7949464d910698593"
format_stamp: "Formatted at 2026-06-19 06:27:56 on dist-test-slave-4t1f"
server_key: "1835aa7914d7f432433e29d273a5ef82"
server_key_iv: "f25a09ee9fb13964d6fb59ea0b602f1f"
server_key_version: "encryptionkey@0"
I20260619 06:27:56.160741 5664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:27:56.163784 5678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:56.164911 5664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:27:56.165158 5664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "85ac1427853048a7949464d910698593"
format_stamp: "Formatted at 2026-06-19 06:27:56 on dist-test-slave-4t1f"
server_key: "1835aa7914d7f432433e29d273a5ef82"
server_key_iv: "f25a09ee9fb13964d6fb59ea0b602f1f"
server_key_version: "encryptionkey@0"
I20260619 06:27:56.165354 5664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:56.210175 5664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:56.211127 5664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:56.211485 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:56.230762 5664 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42203
I20260619 06:27:56.230780 5729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42203 every 8 connection(s)
I20260619 06:27:56.232178 5664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:56.234676 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5664
I20260619 06:27:56.234881 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:56.235634 389 external_mini_cluster.cc:1428] Setting key 321f80533efdde18691403f8598fc5a8
I20260619 06:27:56.236441 5730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:56.248927 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:56.237484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.0.97.126:42203)
0619 06:27:56.238421 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:56.238431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:56.238467 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:27:56.240552 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:56.240576 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:56.240999 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:56.241378 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:56.241898 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.243159 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.244858 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.245509 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.245661 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:56.246822 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:56.246833 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:56.246845 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:56.247405 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:56.247488 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:56.247513 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:56.247590 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:56.247808 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 06:27:56.248097 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":365,"threads_started":1}
I20260619 06:27:56.249312 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593: Bootstrap starting.
I20260619 06:27:56.252269 5730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:56.252437 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:27:56.253435 5730 log.cc:824] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:56.256032 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593: No bootstrap required, opened a new log
I20260619 06:27:56.262688 5730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ac1427853048a7949464d910698593" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42203 } }
I20260619 06:27:56.262970 5730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:56.263041 5730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ac1427853048a7949464d910698593, State: Initialized, Role: FOLLOWER
I20260619 06:27:56.263540 5730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ac1427853048a7949464d910698593" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42203 } }
I20260619 06:27:56.263734 5730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:56.263828 5730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:56.263942 5730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:56.265980 5730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ac1427853048a7949464d910698593" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42203 } }
I20260619 06:27:56.266384 5730 leader_election.cc:302] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85ac1427853048a7949464d910698593; no voters:
I20260619 06:27:56.266937 5730 leader_election.cc:288] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:56.267077 5735 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:56.267760 5735 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [term 1 LEADER]: Becoming Leader. State: Replica: 85ac1427853048a7949464d910698593, State: Running, Role: LEADER
I20260619 06:27:56.268332 5735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ac1427853048a7949464d910698593" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42203 } }
I20260619 06:27:56.269148 5730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:56.271142 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85ac1427853048a7949464d910698593. Latest consensus state: current_term: 1 leader_uuid: "85ac1427853048a7949464d910698593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ac1427853048a7949464d910698593" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42203 } } }
I20260619 06:27:56.271893 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:56.272367 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85ac1427853048a7949464d910698593" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85ac1427853048a7949464d910698593" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42203 } } }
I20260619 06:27:56.272662 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593 [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:56.274266 5744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:56.279004 5744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:56.287905 5744 catalog_manager.cc:1347] Generated new cluster ID: 0814ad50ab7043f6a196d125bbdf7464
I20260619 06:27:56.288021 5744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:56.303503 5744 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:56.304723 5744 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:56.312201 5744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85ac1427853048a7949464d910698593: Generated new TSK 0
I20260619 06:27:56.313107 5744 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 06:27:56.349787 5760 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 06:27:56.349903 5760 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:27:56.350726 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:56.323617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.0.97.126:42203)
0619 06:27:56.323833 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:56.323842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:56.324313 (+ 471us) server_negotiation.cc:407] Connection header received
0619 06:27:56.324524 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:56.324528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:56.324587 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:56.324711 (+ 124us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:27:56.325305 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.326000 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.327915 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.328353 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.328436 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:56.328983 (+ 547us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:27:56.350228 (+ 21245us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:27:56.350497 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54116: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 06:27:56.350874 5733 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54116: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 06:27:56.351058 5759 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:27:56.323576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42203 (local address 127.0.0.1:54116)
0619 06:27:56.324206 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:56.324230 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:56.324369 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:56.324762 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:56.324767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:56.324780 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:27:56.325118 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:56.325130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.326126 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.326133 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:56.327790 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:56.327804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.328500 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.328506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:56.328719 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:56.328763 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:27:56.350438 (+ 21675us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 06:27:56.350684 (+ 246us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:42203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":333,"threads_started":1}
I20260619 06:27:56.352414 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5664
I20260619 06:27:56.353544 5725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:56.442266 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5664
2026-06-19T06:27:56Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (715 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 06:27:56.498061 389 mini_oidc.cc:152] Starting JWKS server
I20260619 06:27:56.499888 389 webserver.cc:460] Webserver started at https://127.0.0.1:41781/ using document root <none> and password file <none>
I20260619 06:27:56.500124 389 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 06:27:56.500739 389 webserver.cc:460] Webserver started at http://127.0.0.1:34219/ using document root <none> and password file <none>
2026-06-19T06:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:56Z Disabled control of system clock
I20260619 06:27:56.526185 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43709
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:36433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43709
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41781/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:27:56.831657 5767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:56.831905 5767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:56.831984 5767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:56.838783 5767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:56.838872 5767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:56.838907 5767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:56.838951 5767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:56.838985 5767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:27:56.839035 5767 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:27:56.839073 5767 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41781/jwks.json
I20260619 06:27:56.847465 5767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41781/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:56.848546 5767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:56.850267 5767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:56.861835 5772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:56.861835 5775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:56.861902 5773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:56.862742 5767 server_base.cc:1034] running on GCE node
I20260619 06:27:56.863390 5767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:56.864559 5767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:56.865741 5767 hybrid_clock.cc:648] HybridClock initialized: now 1781850476865702 us; error 28 us; skew 500 ppm
I20260619 06:27:56.866036 5767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:56.867954 5767 webserver.cc:460] Webserver started at http://127.0.97.126:39261/ using document root <none> and password file <none>
I20260619 06:27:56.868537 5767 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:56.868615 5767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:56.868868 5767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:56.871665 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "da684070988f4d5c9852a8d21923e2cc"
format_stamp: "Formatted at 2026-06-19 06:27:56 on dist-test-slave-4t1f"
server_key: "d7cc13116f4ff1861d8407be2928353b"
server_key_iv: "b846c220b36c30aa7b67a6029976e19f"
server_key_version: "encryptionkey@0"
I20260619 06:27:56.872367 5767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "da684070988f4d5c9852a8d21923e2cc"
format_stamp: "Formatted at 2026-06-19 06:27:56 on dist-test-slave-4t1f"
server_key: "d7cc13116f4ff1861d8407be2928353b"
server_key_iv: "b846c220b36c30aa7b67a6029976e19f"
server_key_version: "encryptionkey@0"
I20260619 06:27:56.876861 5767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 06:27:56.879966 5781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:56.881486 5767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:27:56.881667 5767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "da684070988f4d5c9852a8d21923e2cc"
format_stamp: "Formatted at 2026-06-19 06:27:56 on dist-test-slave-4t1f"
server_key: "d7cc13116f4ff1861d8407be2928353b"
server_key_iv: "b846c220b36c30aa7b67a6029976e19f"
server_key_version: "encryptionkey@0"
I20260619 06:27:56.881850 5767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:56.905628 5767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:56.906457 5767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:56.906738 5767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:56.924646 5767 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43709
I20260619 06:27:56.924669 5832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43709 every 8 connection(s)
I20260619 06:27:56.926031 5767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:56.929698 5833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:56.931924 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5767
I20260619 06:27:56.932142 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:56.932794 389 external_mini_cluster.cc:1428] Setting key fde6393b4565dbac37ae2d9403021f11
I20260619 06:27:56.941237 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc: Bootstrap starting.
I20260619 06:27:56.944654 5833 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:56.945894 5833 log.cc:824] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:56.946589 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:56.934784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.0.97.126:43709)
0619 06:27:56.935766 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:56.935777 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:56.935818 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:27:56.937582 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:56.937626 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:56.938097 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:56.938455 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:56.939042 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.940279 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.942097 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:56.942822 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:56.942968 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:56.944140 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:56.944147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:56.944161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:56.944669 (+ 508us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:56.944761 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:56.944831 (+ 70us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:56.944936 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:56.945228 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:27:56.945536 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":397,"threads_started":1}
I20260619 06:27:56.948554 5833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc: No bootstrap required, opened a new log
I20260619 06:27:56.950093 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:27:56.956634 5833 raft_consensus.cc:348] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da684070988f4d5c9852a8d21923e2cc" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43709 } }
I20260619 06:27:56.956929 5833 raft_consensus.cc:374] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:56.956997 5833 raft_consensus.cc:729] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da684070988f4d5c9852a8d21923e2cc, State: Initialized, Role: FOLLOWER
I20260619 06:27:56.957508 5833 consensus_queue.cc:260] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da684070988f4d5c9852a8d21923e2cc" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43709 } }
I20260619 06:27:56.957726 5833 raft_consensus.cc:388] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:56.957881 5833 raft_consensus.cc:482] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:56.958068 5833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:56.960069 5833 raft_consensus.cc:504] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da684070988f4d5c9852a8d21923e2cc" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43709 } }
I20260619 06:27:56.960527 5833 leader_election.cc:302] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da684070988f4d5c9852a8d21923e2cc; no voters:
I20260619 06:27:56.961041 5833 leader_election.cc:288] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:56.961210 5838 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:56.961915 5838 raft_consensus.cc:686] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [term 1 LEADER]: Becoming Leader. State: Replica: da684070988f4d5c9852a8d21923e2cc, State: Running, Role: LEADER
I20260619 06:27:56.962471 5838 consensus_queue.cc:237] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da684070988f4d5c9852a8d21923e2cc" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43709 } }
I20260619 06:27:56.963011 5833 sys_catalog.cc:564] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:56.965498 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader da684070988f4d5c9852a8d21923e2cc. Latest consensus state: current_term: 1 leader_uuid: "da684070988f4d5c9852a8d21923e2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da684070988f4d5c9852a8d21923e2cc" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43709 } } }
I20260619 06:27:56.965947 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:56.965525 5839 sys_catalog.cc:455] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da684070988f4d5c9852a8d21923e2cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da684070988f4d5c9852a8d21923e2cc" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43709 } } }
I20260619 06:27:56.966734 5839 sys_catalog.cc:458] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc [sys.catalog]: This master's current role is: LEADER
W20260619 06:27:56.974318 5854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:27:56.974459 5854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:27:56.974627 5845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:56.976697 5845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:56.984915 5845 catalog_manager.cc:1347] Generated new cluster ID: fedf65be14c04f4f828a26e72f9517cb
I20260619 06:27:56.985011 5845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:57.000677 5845 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:57.002070 5845 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:57.009341 5845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da684070988f4d5c9852a8d21923e2cc: Generated new TSK 0
I20260619 06:27:57.010231 5845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:27:57.038038 5836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:57.029938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57050 (local address 127.0.97.126:43709)
0619 06:27:57.030120 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:57.030127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:57.030639 (+ 512us) server_negotiation.cc:407] Connection header received
0619 06:27:57.030841 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:57.030847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:57.030939 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:57.031074 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:27:57.031551 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.032648 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.034020 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.034671 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.034789 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:57.035213 (+ 424us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:27:57.037431 (+ 2218us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:27:57.037723 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":64}
W20260619 06:27:57.038160 5836 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57050: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 06:27:57.039758 5862 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:27:57.029921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43709 (local address 127.0.0.1:57050)
0619 06:27:57.030529 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:57.030544 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:57.030659 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:57.031097 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:57.031101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:57.031109 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:27:57.031417 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:57.031425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.032776 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.032781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:57.033898 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:57.033908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.034833 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.034837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:57.034948 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:57.034975 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:27:57.037615 (+ 2640us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 06:27:57.039365 (+ 1750us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:43709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41781/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":1481,"thread_start_us":332,"threads_started":1}
I20260619 06:27:57.040655 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5767
I20260619 06:27:57.041685 5828 generic_service.cc:196] Checking for leaks (request via RPC)
E20260619 06:27:57.050021 5863 webserver.cc:533] Webserver: sslize failed:
E20260619 06:27:57.050133 5863 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 06:27:57.132479 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5767
2026-06-19T06:27:57Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (688 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T06:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:27:57Z Disabled control of system clock
I20260619 06:27:57.210036 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42983
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:35861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42983
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/testchainca.cert with env {}
W20260619 06:27:57.522755 5868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:27:57.523025 5868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:27:57.523159 5868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:27:57.530053 5868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:27:57.530144 5868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:27:57.530243 5868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:27:57.530347 5868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:27:57.530437 5868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 06:27:57.530527 5868 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 06:27:57.530586 5868 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 06:27:57.539858 5868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42983
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:27:57.541061 5868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:27:57.542284 5868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:27:57.553140 5873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:57.553190 5874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:27:57.553239 5876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:27:57.553966 5868 server_base.cc:1034] running on GCE node
I20260619 06:27:57.554749 5868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:27:57.556015 5868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:27:57.557209 5868 hybrid_clock.cc:648] HybridClock initialized: now 1781850477557184 us; error 42 us; skew 500 ppm
I20260619 06:27:57.557528 5868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:27:57.559646 5868 webserver.cc:460] Webserver started at http://127.0.97.126:42197/ using document root <none> and password file <none>
I20260619 06:27:57.560273 5868 fs_manager.cc:359] Metadata directory not provided
I20260619 06:27:57.560411 5868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:27:57.560707 5868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:27:57.563388 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "7bccfc7d0e254073ab0f598259c7be3e"
format_stamp: "Formatted at 2026-06-19 06:27:57 on dist-test-slave-4t1f"
server_key: "bdeb391a415f61fbf9f5e6f10e9ba642"
server_key_iv: "673b0c1b8303294f4ab81461f2cecd77"
server_key_version: "encryptionkey@0"
I20260619 06:27:57.564117 5868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "7bccfc7d0e254073ab0f598259c7be3e"
format_stamp: "Formatted at 2026-06-19 06:27:57 on dist-test-slave-4t1f"
server_key: "bdeb391a415f61fbf9f5e6f10e9ba642"
server_key_iv: "673b0c1b8303294f4ab81461f2cecd77"
server_key_version: "encryptionkey@0"
I20260619 06:27:57.568912 5868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260619 06:27:57.572050 5882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:27:57.573335 5868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:27:57.573513 5868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "7bccfc7d0e254073ab0f598259c7be3e"
format_stamp: "Formatted at 2026-06-19 06:27:57 on dist-test-slave-4t1f"
server_key: "bdeb391a415f61fbf9f5e6f10e9ba642"
server_key_iv: "673b0c1b8303294f4ab81461f2cecd77"
server_key_version: "encryptionkey@0"
I20260619 06:27:57.573710 5868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:27:57.586050 5868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:27:57.586745 5868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:27:57.587000 5868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:27:57.604018 5868 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42983
I20260619 06:27:57.604035 5933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42983 every 8 connection(s)
I20260619 06:27:57.605248 5868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:27:57.606280 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5868
I20260619 06:27:57.606498 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:27:57.607100 389 external_mini_cluster.cc:1428] Setting key 97c113306b754bd1d3dfccdb24b18c68
I20260619 06:27:57.609258 5934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:27:57.621559 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:57.608904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37912 (local address 127.0.97.126:42983)
0619 06:27:57.609929 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:57.609942 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:57.609987 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:27:57.612029 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:57.612054 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:57.612615 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:57.613013 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:27:57.613537 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.615526 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.616781 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.617765 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.617939 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:57.618972 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:27:57.618981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:27:57.618997 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:27:57.619684 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:27:57.619773 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:27:57.619801 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:27:57.619923 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:27:57.620097 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 06:27:57.620409 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":387,"threads_started":1}
I20260619 06:27:57.622499 5934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e: Bootstrap starting.
I20260619 06:27:57.624919 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:27:57.625721 5934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e: Neither blocks nor log segments found. Creating new log.
I20260619 06:27:57.626780 5934 log.cc:824] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:27:57.629441 5934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e: No bootstrap required, opened a new log
I20260619 06:27:57.637212 5934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bccfc7d0e254073ab0f598259c7be3e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42983 } }
I20260619 06:27:57.637669 5934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:27:57.637769 5934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bccfc7d0e254073ab0f598259c7be3e, State: Initialized, Role: FOLLOWER
I20260619 06:27:57.638355 5934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bccfc7d0e254073ab0f598259c7be3e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42983 } }
I20260619 06:27:57.638581 5934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:27:57.638720 5934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:27:57.638899 5934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:27:57.640925 5934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bccfc7d0e254073ab0f598259c7be3e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42983 } }
I20260619 06:27:57.641441 5934 leader_election.cc:302] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bccfc7d0e254073ab0f598259c7be3e; no voters:
I20260619 06:27:57.642012 5934 leader_election.cc:288] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:27:57.642100 5939 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:27:57.642752 5939 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [term 1 LEADER]: Becoming Leader. State: Replica: 7bccfc7d0e254073ab0f598259c7be3e, State: Running, Role: LEADER
I20260619 06:27:57.643266 5939 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bccfc7d0e254073ab0f598259c7be3e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42983 } }
I20260619 06:27:57.643800 5934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:27:57.646088 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bccfc7d0e254073ab0f598259c7be3e. Latest consensus state: current_term: 1 leader_uuid: "7bccfc7d0e254073ab0f598259c7be3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bccfc7d0e254073ab0f598259c7be3e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42983 } } }
I20260619 06:27:57.646502 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:57.647898 5947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:27:57.647869 5940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bccfc7d0e254073ab0f598259c7be3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bccfc7d0e254073ab0f598259c7be3e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42983 } } }
I20260619 06:27:57.648177 5940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e [sys.catalog]: This master's current role is: LEADER
I20260619 06:27:57.656291 5947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:27:57.664366 5947 catalog_manager.cc:1347] Generated new cluster ID: bc15c73d17ac4eaca943ffceb21d5d2a
I20260619 06:27:57.664518 5947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:27:57.675761 5947 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:27:57.676875 5947 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:27:57.682217 5947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bccfc7d0e254073ab0f598259c7be3e: Generated new TSK 0
I20260619 06:27:57.683073 5947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:27:58.177551 5937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:27:57.705591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.0.97.126:42983)
0619 06:27:57.705768 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:27:57.705774 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:27:57.706057 (+ 283us) server_negotiation.cc:407] Connection header received
0619 06:27:57.706252 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:27:57.706259 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:27:57.706341 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:27:57.706509 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0619 06:27:57.706883 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.707959 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.709384 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.710067 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.710189 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:57.710612 (+ 423us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 06:27:58.176949 (+466337us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:27:58.177349 (+ 400us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37916: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:27:58.177632 5937 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37916: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260619 06:27:58.177832 5963 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:27:57.705419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42983 (local address 127.0.0.1:37916)
0619 06:27:57.705964 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 06:27:57.705979 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:27:57.706084 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:27:57.706511 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:27:57.706515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:27:57.706523 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 06:27:57.706743 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:57.706750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.708186 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.708190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:57.709259 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:27:57.709269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:27:57.710211 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:27:57.710215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:27:57.710331 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:27:57.710357 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 06:27:58.177244 (+466887us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 06:27:58.177461 (+ 217us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.126:42983: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":291,"threads_started":1}
I20260619 06:27:58.179001 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5868
I20260619 06:27:58.179993 5929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:27:58.276328 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5868
2026-06-19T06:27:58Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1125 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (403 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (393 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:27:59 dist-test-slave-4t1f krb5kdc[5978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:27:59 dist-test-slave-4t1f krb5kdc[5978](info): set up 2 sockets
Jun 19 06:27:59 dist-test-slave-4t1f krb5kdc[5978](info): commencing operation
krb5kdc: starting...
W20260619 06:28:00.149700 389 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:28:00 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:28:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:28:00.338368 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39557
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:39557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:00.646136 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:00.646499 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:00.646652 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:00.654392 5991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:00.654498 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:00.654541 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:00.654620 5991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:00.654711 5991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:00.663470 5991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:39557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:39557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:00.664642 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:00.665848 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:00.677160 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:00.677111 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:00.677130 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:00.677721 5991 server_base.cc:1034] running on GCE node
I20260619 06:28:00.678455 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:00.679664 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:00.680912 5991 hybrid_clock.cc:648] HybridClock initialized: now 1781850480680833 us; error 52 us; skew 500 ppm
I20260619 06:28:00.681274 5991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:00 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850480, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:00.684266 5991 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:28:00.685698 5991 webserver.cc:460] Webserver started at http://127.0.97.126:33135/ using document root <none> and password file <none>
I20260619 06:28:00.686314 5991 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:00.686451 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:00.686733 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:00.689465 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "394589e921b249df930f74ae0a38d010"
format_stamp: "Formatted at 2026-06-19 06:28:00 on dist-test-slave-4t1f"
server_key: "8c448b73186009f59540778244d42a67"
server_key_iv: "be113c95b89c53b59f4cc9ee9095c861"
server_key_version: "encryptionkey@0"
I20260619 06:28:00.690181 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "394589e921b249df930f74ae0a38d010"
format_stamp: "Formatted at 2026-06-19 06:28:00 on dist-test-slave-4t1f"
server_key: "8c448b73186009f59540778244d42a67"
server_key_iv: "be113c95b89c53b59f4cc9ee9095c861"
server_key_version: "encryptionkey@0"
I20260619 06:28:00.694733 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:00.697902 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:00.699030 5991 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:28:00.699188 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "394589e921b249df930f74ae0a38d010"
format_stamp: "Formatted at 2026-06-19 06:28:00 on dist-test-slave-4t1f"
server_key: "8c448b73186009f59540778244d42a67"
server_key_iv: "be113c95b89c53b59f4cc9ee9095c861"
server_key_version: "encryptionkey@0"
I20260619 06:28:00.699361 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:00.718663 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:00.722709 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:00.723003 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:00.740105 5991 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:39557
I20260619 06:28:00.740144 6057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:39557 every 8 connection(s)
I20260619 06:28:00.741345 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:00.744753 6058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:00.744781 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 5991
I20260619 06:28:00.745007 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:28:00.745575 389 external_mini_cluster.cc:1428] Setting key a66ea159324a23dfbf6a5da86efe004d
I20260619 06:28:00.758193 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010: Bootstrap starting.
Jun 19 06:28:00 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:00.760998 6058 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:00.762431 6058 log.cc:824] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:00.765538 6058 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010: No bootstrap required, opened a new log
I20260619 06:28:00.768847 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:00.747429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45736 (local address 127.0.97.126:39557)
0619 06:28:00.748194 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:00.748207 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:00.748255 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:28:00.750871 (+ 2616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:00.750897 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:00.751366 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:00.751809 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:00.753128 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:00.755133 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:00.756296 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:00.757303 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:00.757468 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:00.761114 (+ 3646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:00.761124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:00.761141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:00.761184 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:00.765004 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:00.765725 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:00.765731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:00.765736 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:00.765908 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:00.766284 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:00.766290 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:00.766293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:00.766750 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:00.767071 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:00.767499 (+ 428us) server_negotiation.cc:299] Negotiation successful
0619 06:28:00.767889 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":278,"threads_started":1}
I20260619 06:28:00.773180 6058 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394589e921b249df930f74ae0a38d010" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39557 } }
I20260619 06:28:00.773525 6058 raft_consensus.cc:374] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:00.773622 6058 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394589e921b249df930f74ae0a38d010, State: Initialized, Role: FOLLOWER
I20260619 06:28:00.774231 6058 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394589e921b249df930f74ae0a38d010" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39557 } }
I20260619 06:28:00.774480 6058 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:00.774616 6058 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:00.774770 6058 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:00.776638 6058 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394589e921b249df930f74ae0a38d010" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39557 } }
I20260619 06:28:00.777148 6058 leader_election.cc:302] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394589e921b249df930f74ae0a38d010; no voters:
I20260619 06:28:00.777700 6058 leader_election.cc:288] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:28:00.777804 6063 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:00.778847 6063 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [term 1 LEADER]: Becoming Leader. State: Replica: 394589e921b249df930f74ae0a38d010, State: Running, Role: LEADER
I20260619 06:28:00.779364 6063 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394589e921b249df930f74ae0a38d010" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39557 } }
I20260619 06:28:00.779749 6058 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:00.782275 6065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394589e921b249df930f74ae0a38d010. Latest consensus state: current_term: 1 leader_uuid: "394589e921b249df930f74ae0a38d010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394589e921b249df930f74ae0a38d010" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39557 } } }
I20260619 06:28:00.782725 6065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:00.782287 6064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "394589e921b249df930f74ae0a38d010" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394589e921b249df930f74ae0a38d010" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 39557 } } }
I20260619 06:28:00.782900 6064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:00.788604 6068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:00.792726 6068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:00.799638 6068 catalog_manager.cc:1347] Generated new cluster ID: dd940b8808e84832bc527c2b83cd3a41
I20260619 06:28:00.799751 6068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:00.814414 6068 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:28:00.816174 6068 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:00.827282 6068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 394589e921b249df930f74ae0a38d010: Generated new TSK 0
I20260619 06:28:00.828280 6068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:28:00.872678 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39557
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:01.183926 6084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:01.184228 6084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:01.184392 6084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:01.191787 6084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:01.191941 6084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:01.192130 6084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:01.201874 6084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:01.203126 6084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:01.204465 6084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:01.217463 6090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:01.217471 6092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:01.218029 6089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:01.218834 6084 server_base.cc:1034] running on GCE node
I20260619 06:28:01.219384 6084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:01.220218 6084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:01.221431 6084 hybrid_clock.cc:648] HybridClock initialized: now 1781850481221376 us; error 62 us; skew 500 ppm
I20260619 06:28:01.221742 6084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:01 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:01.227059 6084 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:28:01.228459 6084 webserver.cc:460] Webserver started at http://127.0.97.65:40157/ using document root <none> and password file <none>
I20260619 06:28:01.229182 6084 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:01.229317 6084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:01.229594 6084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:01.232419 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "007ad9b9c80240288ddb76a92cb1080d"
format_stamp: "Formatted at 2026-06-19 06:28:01 on dist-test-slave-4t1f"
server_key: "835c95324cff70666a147b76e308b3b4"
server_key_iv: "d814c8cd09cdd356ec8e103fec4b391f"
server_key_version: "encryptionkey@0"
I20260619 06:28:01.233225 6084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "007ad9b9c80240288ddb76a92cb1080d"
format_stamp: "Formatted at 2026-06-19 06:28:01 on dist-test-slave-4t1f"
server_key: "835c95324cff70666a147b76e308b3b4"
server_key_iv: "d814c8cd09cdd356ec8e103fec4b391f"
server_key_version: "encryptionkey@0"
I20260619 06:28:01.237931 6084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:28:01.241189 6099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:01.242360 6084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:28:01.242533 6084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "007ad9b9c80240288ddb76a92cb1080d"
format_stamp: "Formatted at 2026-06-19 06:28:01 on dist-test-slave-4t1f"
server_key: "835c95324cff70666a147b76e308b3b4"
server_key_iv: "d814c8cd09cdd356ec8e103fec4b391f"
server_key_version: "encryptionkey@0"
I20260619 06:28:01.242729 6084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:01.256201 6084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:01.259683 6084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:01.259954 6084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:01.260807 6084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:01.262038 6084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:01.262141 6084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:01.262303 6084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:01.262375 6084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:01.294543 6084 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:45485
I20260619 06:28:01.294582 6211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:45485 every 8 connection(s)
I20260619 06:28:01.295770 6084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:01.302393 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 6084
I20260619 06:28:01.302709 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:28:01.303365 389 external_mini_cluster.cc:1428] Setting key a976bf1866d55a4c403e515cc922999e
Jun 19 06:28:01 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:01.316596 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:01.298199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34367 (local address 127.0.97.126:39557)
0619 06:28:01.298630 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:01.298634 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:01.299532 (+ 898us) server_negotiation.cc:407] Connection header received
0619 06:28:01.301308 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:01.301314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:01.301405 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:01.301519 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:01.303178 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.304084 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.306134 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.306846 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.307009 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:01.310883 (+ 3874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:01.310891 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:01.310898 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:01.310937 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:01.313570 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:01.314447 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:01.314451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:01.314453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:01.314563 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:01.315021 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:01.315024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:01.315026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:01.315264 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:01.315366 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:01.316164 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 06:28:01.316315 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":230,"threads_started":1}
I20260619 06:28:01.318151 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:01.298627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39557 (local address 127.0.97.65:34367)
0619 06:28:01.299361 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:01.299409 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:01.300993 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:01.301754 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:01.301762 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:01.302252 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:01.302994 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:01.303012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.304223 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.304227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:01.305986 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:01.305998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.307037 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.307042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:01.307226 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:01.307888 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:01.307907 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:01.310683 (+ 2776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:01.313691 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:01.313703 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:01.313720 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:01.314335 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:01.314665 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:01.314673 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:01.314679 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:01.314913 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:01.315379 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:01.315389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:01.315809 (+ 420us) client_negotiation.cc:769] Sending connection context
0619 06:28:01.316098 (+ 289us) client_negotiation.cc:240] Negotiation successful
0619 06:28:01.316672 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":302,"threads_started":1}
I20260619 06:28:01.319730 6212 heartbeater.cc:347] Connected to a master server at 127.0.97.126:39557
I20260619 06:28:01.320010 6212 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:01.320775 6212 heartbeater.cc:511] Master 127.0.97.126:39557 requested a full tablet report, sending...
I20260619 06:28:01.322758 6023 ts_manager.cc:194] Registered new tserver with Master: 007ad9b9c80240288ddb76a92cb1080d (127.0.97.65:45485)
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260619 06:28:01.324215 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:34367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:28:01.347712 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39557
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:01.655718 6218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:01.655956 6218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:01.656137 6218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:01.662822 6218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:01.662936 6218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:01.663110 6218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:01.672207 6218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:01.673501 6218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:01.674726 6218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:01.687121 6226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:01.687196 6223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:01.687547 6224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:01.687885 6218 server_base.cc:1034] running on GCE node
I20260619 06:28:01.688432 6218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:01.689361 6218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:01.690546 6218 hybrid_clock.cc:648] HybridClock initialized: now 1781850481690509 us; error 31 us; skew 500 ppm
I20260619 06:28:01.690834 6218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:01 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:01.693904 6218 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:28:01.695279 6218 webserver.cc:460] Webserver started at http://127.0.97.66:42125/ using document root <none> and password file <none>
I20260619 06:28:01.695936 6218 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:01.696080 6218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:01.696362 6218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:01.699205 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "ebe61d681148478890ff11f2471485ed"
format_stamp: "Formatted at 2026-06-19 06:28:01 on dist-test-slave-4t1f"
server_key: "97bfac0dd792d2237dae97b0b9152b71"
server_key_iv: "6ad547ac428a0b9a84e0e7ad5321021f"
server_key_version: "encryptionkey@0"
I20260619 06:28:01.699937 6218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "ebe61d681148478890ff11f2471485ed"
format_stamp: "Formatted at 2026-06-19 06:28:01 on dist-test-slave-4t1f"
server_key: "97bfac0dd792d2237dae97b0b9152b71"
server_key_iv: "6ad547ac428a0b9a84e0e7ad5321021f"
server_key_version: "encryptionkey@0"
I20260619 06:28:01.704942 6218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:28:01.708065 6233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:01.709293 6218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:28:01.709441 6218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "ebe61d681148478890ff11f2471485ed"
format_stamp: "Formatted at 2026-06-19 06:28:01 on dist-test-slave-4t1f"
server_key: "97bfac0dd792d2237dae97b0b9152b71"
server_key_iv: "6ad547ac428a0b9a84e0e7ad5321021f"
server_key_version: "encryptionkey@0"
I20260619 06:28:01.709625 6218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:01.721721 6218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:01.725379 6218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:01.725646 6218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:01.726507 6218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:01.727725 6218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:01.727828 6218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:01.727967 6218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:01.728040 6218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:01.758782 6218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:36479
I20260619 06:28:01.758801 6345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:36479 every 8 connection(s)
I20260619 06:28:01.760300 6218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:01.763955 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 6218
I20260619 06:28:01.764168 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:28:01.764696 389 external_mini_cluster.cc:1428] Setting key bd958627fdb8f8095784bd9a933f015b
Jun 19 06:28:01 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850481, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:01.780041 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:01.762917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56793 (local address 127.0.97.126:39557)
0619 06:28:01.763089 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:01.763094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:01.764548 (+ 1454us) server_negotiation.cc:407] Connection header received
0619 06:28:01.766524 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:01.766529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:01.766619 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:01.766728 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:01.768134 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.768813 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.770406 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.770816 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.770948 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:01.774639 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:01.774645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:01.774650 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:01.774708 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:01.777163 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:01.777845 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:01.777850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:01.777853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:01.777984 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:01.778411 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:01.778415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:01.778419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:01.778660 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:01.778802 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:01.779445 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 06:28:01.779685 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:28:01.781236 6348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:01.763466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39557 (local address 127.0.97.66:56793)
0619 06:28:01.764383 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:01.764437 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:01.766264 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:01.766949 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:01.766958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:01.767431 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:01.767969 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:01.767985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.768972 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.768977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:01.770251 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:01.770263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:01.771029 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:01.771036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:01.771307 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:01.772036 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:01.772068 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:01.774421 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:01.777314 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:01.777322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:01.777337 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:01.777724 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:01.778111 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:01.778116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:01.778120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:01.778296 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:01.778789 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:01.778796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:01.779106 (+ 310us) client_negotiation.cc:769] Sending connection context
0619 06:28:01.779360 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 06:28:01.779918 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":357,"threads_started":1}
I20260619 06:28:01.782564 6346 heartbeater.cc:347] Connected to a master server at 127.0.97.126:39557
I20260619 06:28:01.782841 6346 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260619 06:28:01.783541 6346 heartbeater.cc:511] Master 127.0.97.126:39557 requested a full tablet report, sending...
I20260619 06:28:01.784857 6023 ts_manager.cc:194] Registered new tserver with Master: ebe61d681148478890ff11f2471485ed (127.0.97.66:36479)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260619 06:28:01.786039 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:56793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:28:01.806648 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39557
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:44625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:02.122975 6351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:02.123411 6351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:02.123611 6351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:02.132761 6351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:02.132864 6351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:02.133050 6351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:02.142598 6351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:39557
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:02.143824 6351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:02.145195 6351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:02.158051 6357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:02.158068 6356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:02.158102 6359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:02.159162 6351 server_base.cc:1034] running on GCE node
I20260619 06:28:02.159668 6351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:02.160470 6351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:02.161679 6351 hybrid_clock.cc:648] HybridClock initialized: now 1781850482161646 us; error 36 us; skew 500 ppm
I20260619 06:28:02.161957 6351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.164906 6351 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:28:02.166221 6351 webserver.cc:460] Webserver started at http://127.0.97.67:35527/ using document root <none> and password file <none>
I20260619 06:28:02.166846 6351 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:02.166985 6351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:02.167268 6351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:02.169994 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "6a0b9cae086146b284579ea6a89784b5"
format_stamp: "Formatted at 2026-06-19 06:28:02 on dist-test-slave-4t1f"
server_key: "23f5bbdd43e8e4fddfe7aa4f51751af2"
server_key_iv: "c3addad962afec1d57a9f6451b9b6e55"
server_key_version: "encryptionkey@0"
I20260619 06:28:02.170732 6351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "6a0b9cae086146b284579ea6a89784b5"
format_stamp: "Formatted at 2026-06-19 06:28:02 on dist-test-slave-4t1f"
server_key: "23f5bbdd43e8e4fddfe7aa4f51751af2"
server_key_iv: "c3addad962afec1d57a9f6451b9b6e55"
server_key_version: "encryptionkey@0"
I20260619 06:28:02.175355 6351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:02.178542 6366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:02.179847 6351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:28:02.180003 6351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "6a0b9cae086146b284579ea6a89784b5"
format_stamp: "Formatted at 2026-06-19 06:28:02 on dist-test-slave-4t1f"
server_key: "23f5bbdd43e8e4fddfe7aa4f51751af2"
server_key_iv: "c3addad962afec1d57a9f6451b9b6e55"
server_key_version: "encryptionkey@0"
I20260619 06:28:02.180219 6351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:02.210256 6351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:02.214479 6351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:02.214756 6351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:02.215643 6351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:02.216876 6351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:02.216950 6351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:02.217131 6351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:02.217216 6351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260619 06:28:02.247340 6351 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:33017
I20260619 06:28:02.247392 6478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:33017 every 8 connection(s)
I20260619 06:28:02.248555 6351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:02.255115 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 6351
I20260619 06:28:02.255327 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:28:02.255708 389 external_mini_cluster.cc:1428] Setting key 09df91f769c2ced7f5cd80657b5f30d8
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:02.267127 6214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.251159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:40029 (local address 127.0.97.126:39557)
0619 06:28:02.251327 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.251332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.252432 (+ 1100us) server_negotiation.cc:407] Connection header received
0619 06:28:02.254156 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.254161 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.254245 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.254352 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.255832 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.256700 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.258222 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.258633 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.258762 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.261958 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:02.261963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:02.261966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:02.261994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:02.264342 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.265008 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.265013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.265017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.265198 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.265598 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.265602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.265606 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.265853 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:02.265998 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:02.266602 (+ 604us) server_negotiation.cc:299] Negotiation successful
0619 06:28:02.266810 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:28:02.268127 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.251619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:39557 (local address 127.0.97.67:40029)
0619 06:28:02.252282 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:02.252318 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:02.253888 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:02.254572 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:02.254580 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:02.255057 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:02.255678 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:02.255695 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.256844 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.256853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:02.258083 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:02.258095 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.258835 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.258844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:02.259076 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.259718 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:02.259739 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:02.261763 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:02.264478 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:02.264486 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:02.264500 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:02.264887 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:02.265302 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:02.265308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:02.265312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:02.265483 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:02.265971 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:02.265979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:02.266281 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 06:28:02.266516 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 06:28:02.267009 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":274,"threads_started":1}
I20260619 06:28:02.269695 6479 heartbeater.cc:347] Connected to a master server at 127.0.97.126:39557
I20260619 06:28:02.270005 6479 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:02.270825 6479 heartbeater.cc:511] Master 127.0.97.126:39557 requested a full tablet report, sending...
I20260619 06:28:02.272163 6023 ts_manager.cc:194] Registered new tserver with Master: 6a0b9cae086146b284579ea6a89784b5 (127.0.97.67:33017)
I20260619 06:28:02.272939 6023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:40029
I20260619 06:28:02.283455 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/test-admin.keytab.
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.309618 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 06:28:02.314461 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:02.312044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44876 (local address 127.0.97.65:45485)
0619 06:28:02.312649 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.312657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.312684 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:28:02.312741 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.312747 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.312945 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.313159 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.314022 (+ 863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44876: BlockingRecv error: recv got EOF from 127.0.0.1:44876 (error 108)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":295,"threads_started":1}
W20260619 06:28:02.314535 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.316107 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.317479 389 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:28:02.327210 6212 heartbeater.cc:503] Master 127.0.97.126:39557 was elected leader, sending a full tablet report...
I20260619 06:28:02.332496 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.319043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44878 (local address 127.0.97.65:45485)
0619 06:28:02.319234 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.319241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.319473 (+ 232us) server_negotiation.cc:407] Connection header received
0619 06:28:02.319683 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.319688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.319753 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.319853 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.320755 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.321795 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.322995 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.323554 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.323688 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.327209 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:02.327219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:02.327232 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:02.327261 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:02.330322 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.330904 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.330911 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.330913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.331023 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.331332 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.331335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.331336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.331638 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:02.331847 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:02.332061 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 06:28:02.332258 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 06:28:02.337894 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:02.336266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.0.97.65:45485)
0619 06:28:02.336457 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.336463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.336636 (+ 173us) server_negotiation.cc:407] Connection header received
0619 06:28:02.336819 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.336824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.336893 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.336988 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.337666 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44884: BlockingRecv error: recv got EOF from 127.0.0.1:44884 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:28:02.338076 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.339470 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.340651 389 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:28:02.353551 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.341956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.0.97.65:45485)
0619 06:28:02.342115 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.342123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.342259 (+ 136us) server_negotiation.cc:407] Connection header received
0619 06:28:02.342353 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.342357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.342401 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.342472 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.343275 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.343930 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.345021 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.345465 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.345562 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.348973 (+ 3411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:02.348978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:02.348981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:02.349016 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:02.351193 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.351825 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.351832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.351840 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.351955 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.352333 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.352340 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.352343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.352576 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:02.352835 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:02.353127 (+ 292us) server_negotiation.cc:299] Negotiation successful
0619 06:28:02.353308 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 06:28:02.357213 6486 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:02.355703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.0.97.65:45485)
0619 06:28:02.355830 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.355834 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.356205 (+ 371us) server_negotiation.cc:407] Connection header received
0619 06:28:02.356303 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.356307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.356363 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.356434 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.356961 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44896: BlockingRecv error: recv got EOF from 127.0.0.1:44896 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260619 06:28:02.357411 389 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781850433974487-389-0/krb5kdc/krb5cc' of type FILE
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.358707 389 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:02.359903 389 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:28:02.371965 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.360991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44912 (local address 127.0.97.65:45485)
0619 06:28:02.361194 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.361199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.361359 (+ 160us) server_negotiation.cc:407] Connection header received
0619 06:28:02.361453 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.361456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.361500 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.361569 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.362374 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.363022 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.364115 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.364485 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.364568 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.367656 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:02.367661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:02.367664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:02.367688 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:02.369901 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.370452 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.370455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.370457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.370581 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.370912 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.370915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.370917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.371142 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:02.371341 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:02.371513 (+ 172us) server_negotiation.cc:299] Negotiation successful
0619 06:28:02.371689 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 06:28:02.373322 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 6084
I20260619 06:28:02.384687 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.373938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.0.97.65:45485)
0619 06:28:02.374057 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.374060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.374449 (+ 389us) server_negotiation.cc:407] Connection header received
0619 06:28:02.374547 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.374551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.374604 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.374690 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.375450 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.376150 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.377374 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.377830 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.377931 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.379799 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:02.379805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:02.379808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:02.379835 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:02.382514 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.383069 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.383073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.383075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.383195 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.383582 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.383585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.383587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.383802 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:02.384065 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:02.384241 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 06:28:02.384387 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:28:02.385251 6207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:02.466058 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 6084
I20260619 06:28:02.482453 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 6218
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:28:02.497182 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.483344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.97.66:36479)
0619 06:28:02.483905 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.483912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.483937 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:28:02.483994 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.483999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.484267 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.484405 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.485389 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.486385 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.487627 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.488192 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.488331 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.491965 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:02.491974 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:02.491987 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:02.492016 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:02.494839 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.495434 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.495441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.495443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.495548 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.495869 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.495872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.495874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.496163 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:02.496390 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:02.496656 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 06:28:02.496851 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":268,"threads_started":1}
I20260619 06:28:02.498435 6341 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:02.579195 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 6218
I20260619 06:28:02.594741 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 6351
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[5978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:28:02.609483 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:02.595703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.0.97.67:33017)
0619 06:28:02.596325 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:02.596332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:02.596360 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:28:02.596416 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:02.596422 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:02.596627 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:02.596756 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:02.597743 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.598726 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.599852 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:02.600393 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:02.600485 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:02.604262 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:02.604272 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:02.604285 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:02.604314 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:02.607209 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.607771 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.607778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.607780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.607888 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:02.608172 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:02.608175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:02.608177 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:02.608476 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:02.608759 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:02.608915 (+ 156us) server_negotiation.cc:299] Negotiation successful
0619 06:28:02.609172 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":328,"threads_started":1}
I20260619 06:28:02.610683 6474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:02.692672 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 6351
I20260619 06:28:02.708050 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 5991
I20260619 06:28:02.709220 6053 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:02.808316 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 5991
2026-06-19T06:28:02Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3735 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[6506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[6506](info): set up 2 sockets
Jun 19 06:28:02 dist-test-slave-4t1f krb5kdc[6506](info): commencing operation
krb5kdc: starting...
W20260619 06:28:03.881773 389 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:28:04 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab.
I20260619 06:28:04.072819 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38673
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:04.385007 6519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:04.385346 6519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:04.385434 6519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:04.392311 6519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:04.392403 6519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:04.392448 6519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:04.392504 6519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:04.392561 6519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:04.401769 6519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:04.402944 6519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:04.404134 6519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:04.415221 6524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:04.415258 6527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:04.415295 6525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:04.416420 6519 server_base.cc:1034] running on GCE node
I20260619 06:28:04.417181 6519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:04.418462 6519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:04.419703 6519 hybrid_clock.cc:648] HybridClock initialized: now 1781850484419669 us; error 52 us; skew 500 ppm
I20260619 06:28:04.420027 6519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:04 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:04.422973 6519 init.cc:377] Logged in from keytab as oryx/127.0.97.126@KRBTEST.COM (short username oryx)
I20260619 06:28:04.424281 6519 webserver.cc:460] Webserver started at http://127.0.97.126:37531/ using document root <none> and password file <none>
I20260619 06:28:04.424912 6519 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:04.425055 6519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:04.425408 6519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:04.428184 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "682d9d818385488cb67aa771b5305d6a"
format_stamp: "Formatted at 2026-06-19 06:28:04 on dist-test-slave-4t1f"
server_key: "e13d472b267c9c626186fa2d7c039c7e"
server_key_iv: "6216dc87836eae83e78e961fada03fbd"
server_key_version: "encryptionkey@0"
I20260619 06:28:04.428921 6519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "682d9d818385488cb67aa771b5305d6a"
format_stamp: "Formatted at 2026-06-19 06:28:04 on dist-test-slave-4t1f"
server_key: "e13d472b267c9c626186fa2d7c039c7e"
server_key_iv: "6216dc87836eae83e78e961fada03fbd"
server_key_version: "encryptionkey@0"
I20260619 06:28:04.433759 6519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:28:04.436839 6534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:04.438109 6519 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:28:04.438278 6519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "682d9d818385488cb67aa771b5305d6a"
format_stamp: "Formatted at 2026-06-19 06:28:04 on dist-test-slave-4t1f"
server_key: "e13d472b267c9c626186fa2d7c039c7e"
server_key_iv: "6216dc87836eae83e78e961fada03fbd"
server_key_version: "encryptionkey@0"
I20260619 06:28:04.438462 6519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:04.460006 6519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:04.463711 6519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:04.463977 6519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:04.481550 6519 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38673
I20260619 06:28:04.481585 6585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38673 every 8 connection(s)
I20260619 06:28:04.482789 6519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:04.483973 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 6519
I20260619 06:28:04.484202 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:28:04.484989 389 external_mini_cluster.cc:1428] Setting key cb176d010c56b6484bacd0075629b654
I20260619 06:28:04.487329 6586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:28:04 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:04.500041 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: Bootstrap starting.
I20260619 06:28:04.503032 6586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:04.504030 6586 log.cc:824] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:04.506824 6586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: No bootstrap required, opened a new log
I20260619 06:28:04.507378 6589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:04.486963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41640 (local address 127.0.97.126:38673)
0619 06:28:04.487783 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:04.487793 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:04.487831 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:28:04.489438 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:04.489458 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:04.489818 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:04.490148 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:04.491835 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:04.493205 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:04.494932 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:04.495556 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:04.495684 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:04.500283 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:04.500295 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:04.500309 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:04.500343 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:04.503369 (+ 3026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:04.504050 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:04.504060 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:04.504067 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:04.504285 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:04.504677 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:04.504683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:04.504687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:04.505283 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:04.505586 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:04.506016 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 06:28:04.506410 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":413,"threads_started":1}
I20260619 06:28:04.514186 6586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } }
I20260619 06:28:04.514596 6586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:04.514699 6586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682d9d818385488cb67aa771b5305d6a, State: Initialized, Role: FOLLOWER
I20260619 06:28:04.515340 6586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } }
I20260619 06:28:04.515620 6586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:04.515774 6586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:04.515950 6586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:04.518146 6586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } }
I20260619 06:28:04.518602 6586 leader_election.cc:302] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 682d9d818385488cb67aa771b5305d6a; no voters:
I20260619 06:28:04.519294 6586 leader_election.cc:288] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:28:04.519428 6591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:04.520138 6591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 1 LEADER]: Becoming Leader. State: Replica: 682d9d818385488cb67aa771b5305d6a, State: Running, Role: LEADER
I20260619 06:28:04.520676 6591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } }
I20260619 06:28:04.521399 6586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:04.523414 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 682d9d818385488cb67aa771b5305d6a. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } }
I20260619 06:28:04.523502 6592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } }
I20260619 06:28:04.523785 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:04.523785 6592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:04.524883 6600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:04.530140 6600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:04.538334 6600 catalog_manager.cc:1347] Generated new cluster ID: 53557a40c9e54368b7ff1a27442b6eb3
I20260619 06:28:04.538442 6600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:04.551086 6600 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:28:04.552795 6600 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:04.573988 6600 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: Generated new TSK 0
I20260619 06:28:04.574923 6600 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260619 06:28:04.632661 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38673
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:04.946281 6612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:04.946530 6612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:04.946727 6612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:04.953779 6612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:04.953918 6612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:04.954113 6612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:04.963282 6612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:04.964426 6612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:04.965654 6612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:04.978259 6618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:04.978255 6620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:04.978294 6617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:04.979121 6612 server_base.cc:1034] running on GCE node
I20260619 06:28:04.979645 6612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:04.980544 6612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:04.981752 6612 hybrid_clock.cc:648] HybridClock initialized: now 1781850484981674 us; error 78 us; skew 500 ppm
I20260619 06:28:04.982043 6612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:04 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:04.984925 6612 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260619 06:28:04.986259 6612 webserver.cc:460] Webserver started at http://127.0.97.65:36891/ using document root <none> and password file <none>
I20260619 06:28:04.986865 6612 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:04.987004 6612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:04.987273 6612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:04.989919 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "22195c66c2644992a4847fa8c008a276"
format_stamp: "Formatted at 2026-06-19 06:28:04 on dist-test-slave-4t1f"
server_key: "66afe5b13ec87c12c830b7e08419d028"
server_key_iv: "d5293639c5abcf35d74d9c2df99c680a"
server_key_version: "encryptionkey@0"
I20260619 06:28:04.990653 6612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "22195c66c2644992a4847fa8c008a276"
format_stamp: "Formatted at 2026-06-19 06:28:04 on dist-test-slave-4t1f"
server_key: "66afe5b13ec87c12c830b7e08419d028"
server_key_iv: "d5293639c5abcf35d74d9c2df99c680a"
server_key_version: "encryptionkey@0"
I20260619 06:28:04.995355 6612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:04.998590 6627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:04.999742 6612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:28:04.999909 6612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "22195c66c2644992a4847fa8c008a276"
format_stamp: "Formatted at 2026-06-19 06:28:04 on dist-test-slave-4t1f"
server_key: "66afe5b13ec87c12c830b7e08419d028"
server_key_iv: "d5293639c5abcf35d74d9c2df99c680a"
server_key_version: "encryptionkey@0"
I20260619 06:28:05.000116 6612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:05.038847 6612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:05.042886 6612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:05.043176 6612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:05.044055 6612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:05.045384 6612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:05.045480 6612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:05.045641 6612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:05.045709 6612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:05.076895 6612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44529
I20260619 06:28:05.076974 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44529 every 8 connection(s)
I20260619 06:28:05.078147 6612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:05.082204 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 6612
I20260619 06:28:05.082409 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:28:05.082995 389 external_mini_cluster.cc:1428] Setting key 4c85cf9b14e25638e21a9dcaae33fa02
Jun 19 06:28:05 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:05.100019 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:05.080531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42483 (local address 127.0.97.126:38673)
0619 06:28:05.080961 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:05.080965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:05.081847 (+ 882us) server_negotiation.cc:407] Connection header received
0619 06:28:05.083908 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:05.083915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:05.083993 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:05.084104 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:05.086004 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.086706 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.089124 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.089492 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.089592 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:05.093303 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:05.093312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:05.093320 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:05.093365 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:05.096825 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:05.097536 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:05.097545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:05.097552 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:05.097715 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:05.098078 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:05.098086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:05.098092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:05.098455 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:05.098602 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:05.099251 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 06:28:05.099500 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":342,"thread_start_us":225,"threads_started":1}
I20260619 06:28:05.100579 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:05.080966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:42483)
0619 06:28:05.081687 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:05.081729 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:05.083462 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:05.084294 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:05.084302 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:05.085022 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:05.085839 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:05.085865 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.086857 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.086866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:05.088942 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:05.088959 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.089709 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.089718 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:05.090034 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:05.090735 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:05.090782 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:05.093059 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:05.096943 (+ 3884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:05.096949 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:05.096962 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:05.097409 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:05.097820 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:05.097823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:05.097825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:05.097976 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:05.098570 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:05.098575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:05.098850 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 06:28:05.099071 (+ 221us) client_negotiation.cc:240] Negotiation successful
0619 06:28:05.099413 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":259,"threads_started":1}
I20260619 06:28:05.102085 6740 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38673
I20260619 06:28:05.102421 6740 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:05.103341 6740 heartbeater.cc:511] Master 127.0.97.126:38673 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260619 06:28:05.105620 6551 ts_manager.cc:194] Registered new tserver with Master: 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
I20260619 06:28:05.107146 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:42483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab.
I20260619 06:28:05.128014 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38673
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:05.440521 6746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:05.440830 6746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:05.440990 6746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:05.448220 6746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:05.448355 6746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:05.448520 6746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:05.457754 6746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:05.458909 6746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:05.460150 6746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:05.472412 6754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:05.472358 6752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:05.472514 6751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:05.473238 6746 server_base.cc:1034] running on GCE node
I20260619 06:28:05.473788 6746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:05.474620 6746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:05.475867 6746 hybrid_clock.cc:648] HybridClock initialized: now 1781850485475822 us; error 47 us; skew 500 ppm
I20260619 06:28:05.476209 6746 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:05 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850485, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:05.479127 6746 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260619 06:28:05.480448 6746 webserver.cc:460] Webserver started at http://127.0.97.66:33721/ using document root <none> and password file <none>
I20260619 06:28:05.481166 6746 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:05.481314 6746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:05.481588 6746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:05.484223 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "7c6e557d99c841e7b8b2ca2eabac1616"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "b66a1ddfaefccd2b7236fa0ef24e8eb2"
server_key_iv: "1bb7c3ce9da62e717055a91c5447906f"
server_key_version: "encryptionkey@0"
I20260619 06:28:05.484968 6746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "7c6e557d99c841e7b8b2ca2eabac1616"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "b66a1ddfaefccd2b7236fa0ef24e8eb2"
server_key_iv: "1bb7c3ce9da62e717055a91c5447906f"
server_key_version: "encryptionkey@0"
I20260619 06:28:05.489935 6746 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 06:28:05.493391 6761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:05.494652 6746 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:28:05.494814 6746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "7c6e557d99c841e7b8b2ca2eabac1616"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "b66a1ddfaefccd2b7236fa0ef24e8eb2"
server_key_iv: "1bb7c3ce9da62e717055a91c5447906f"
server_key_version: "encryptionkey@0"
I20260619 06:28:05.494993 6746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:05.519598 6746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:05.524055 6746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:05.524336 6746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:05.525331 6746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:05.526597 6746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:05.526708 6746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:05.526851 6746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:05.526921 6746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:05.558575 6746 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:42923
I20260619 06:28:05.558648 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:42923 every 8 connection(s)
I20260619 06:28:05.559806 6746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:05.564559 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 6746
I20260619 06:28:05.564745 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:28:05.565183 389 external_mini_cluster.cc:1428] Setting key 9c4037f584d6e701581cd024d864a498
Jun 19 06:28:05 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850485, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:05.579383 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:05.562208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:48781 (local address 127.0.97.126:38673)
0619 06:28:05.562382 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:05.562386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:05.563409 (+ 1023us) server_negotiation.cc:407] Connection header received
0619 06:28:05.565207 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:05.565212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:05.565295 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:05.565395 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:05.567049 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.567766 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.569295 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.569703 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.569827 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:05.573027 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:05.573036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:05.573044 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:05.573115 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:05.576490 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:05.577089 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:05.577097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:05.577103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:05.577250 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:05.577619 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:05.577627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:05.577633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:05.578008 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:05.578153 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:05.578709 (+ 556us) server_negotiation.cc:299] Negotiation successful
0619 06:28:05.578944 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:28:05.579964 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:05.562616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:48781)
0619 06:28:05.563246 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:05.563286 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:05.564920 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:05.565626 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:05.565634 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:05.566169 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:05.566883 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:05.566901 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.567902 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.567912 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:05.569154 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:05.569166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:05.569904 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:05.569911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:05.570138 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:05.570751 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:05.570771 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:05.572814 (+ 2043us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:05.576610 (+ 3796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:05.576616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:05.576626 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:05.576964 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:05.577339 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:05.577342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:05.577344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:05.577512 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:05.578120 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:05.578125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:05.578377 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 06:28:05.578577 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 06:28:05.578908 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":244,"threads_started":1}
I20260619 06:28:05.581370 6874 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38673
I20260619 06:28:05.581660 6874 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:05.582293 6874 heartbeater.cc:511] Master 127.0.97.126:38673 requested a full tablet report, sending...
I20260619 06:28:05.583667 6551 ts_manager.cc:194] Registered new tserver with Master: 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923)
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260619 06:28:05.584496 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:48781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260619 06:28:05.608136 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38673
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:05.916143 6879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:05.916404 6879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:05.916491 6879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:05.923519 6879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:05.923615 6879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:05.923727 6879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:05.933467 6879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:05.934908 6879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:05.936235 6879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:05.949000 6884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:05.949095 6885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:05.948978 6887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:05.949684 6879 server_base.cc:1034] running on GCE node
I20260619 06:28:05.950196 6879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:05.950966 6879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:05.952179 6879 hybrid_clock.cc:648] HybridClock initialized: now 1781850485952143 us; error 43 us; skew 500 ppm
I20260619 06:28:05.952469 6879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:05 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850485, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:05.955480 6879 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260619 06:28:05.956830 6879 webserver.cc:460] Webserver started at http://127.0.97.67:45811/ using document root <none> and password file <none>
I20260619 06:28:05.957508 6879 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:05.957643 6879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:05.957911 6879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:05.960546 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "dc91eac4703c451a88feb1b940ed7833"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "54ee7c44203448cc11bb1d8620fef89d"
server_key_iv: "ac15a5bd806067605c055ace2a08c87f"
server_key_version: "encryptionkey@0"
I20260619 06:28:05.961372 6879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "dc91eac4703c451a88feb1b940ed7833"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "54ee7c44203448cc11bb1d8620fef89d"
server_key_iv: "ac15a5bd806067605c055ace2a08c87f"
server_key_version: "encryptionkey@0"
I20260619 06:28:05.966053 6879 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:28:05.969215 6894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:05.970383 6879 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:28:05.970534 6879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "dc91eac4703c451a88feb1b940ed7833"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "54ee7c44203448cc11bb1d8620fef89d"
server_key_iv: "ac15a5bd806067605c055ace2a08c87f"
server_key_version: "encryptionkey@0"
I20260619 06:28:05.970711 6879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:05.990103 6879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:05.993810 6879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:05.994088 6879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:05.994880 6879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:05.996031 6879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:05.996117 6879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:05.996274 6879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:05.996321 6879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:06.025769 6879 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39745
I20260619 06:28:06.025779 7006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39745 every 8 connection(s)
I20260619 06:28:06.027031 6879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:06.035305 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 6879
I20260619 06:28:06.035482 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:28:06.035993 389 external_mini_cluster.cc:1428] Setting key 7ec4566e0a1e62e63b9137ac0ad4d2b7
Jun 19 06:28:06 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850485, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:06.046039 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.029386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51285 (local address 127.0.97.126:38673)
0619 06:28:06.029596 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.029602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.030732 (+ 1130us) server_negotiation.cc:407] Connection header received
0619 06:28:06.032401 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.032406 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.032470 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.032555 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.033986 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.034779 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.036939 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.037521 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.037613 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.040996 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.041000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.041004 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.041033 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.043678 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.044250 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.044254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.044257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.044378 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.044726 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.044729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.044730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.044977 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.045143 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.045642 (+ 499us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.045806 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:28:06.046864 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.029916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:51285)
0619 06:28:06.030595 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:06.030629 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:06.032174 (+ 1545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:06.032740 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:06.032747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:06.033317 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:06.033839 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.033855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.034931 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.034935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.036793 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.036808 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.037733 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.037738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.038009 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.038710 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:06.038743 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:06.040839 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:06.043777 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.043783 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.043792 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.044141 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.044461 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.044464 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.044466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.044630 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.045120 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:06.045125 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:06.045377 (+ 252us) client_negotiation.cc:769] Sending connection context
0619 06:28:06.045578 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 06:28:06.045905 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":277,"threads_started":1}
I20260619 06:28:06.047940 7007 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38673
I20260619 06:28:06.048195 7007 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:06.048785 7007 heartbeater.cc:511] Master 127.0.97.126:38673 requested a full tablet report, sending...
I20260619 06:28:06.050347 6551 ts_manager.cc:194] Registered new tserver with Master: dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745)
I20260619 06:28:06.050913 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:28:06.051223 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:51285
WARNING: no policy specified for oryx/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.125.keytab.
Entry for principal oryx/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.125.keytab.
I20260619 06:28:06.096661 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:43235
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.125.keytab
--principal=oryx/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38673,127.0.97.125:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:28:06.109908 6740 heartbeater.cc:503] Master 127.0.97.126:38673 was elected leader, sending a full tablet report...
W20260619 06:28:06.408494 7012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:06.408818 7012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:06.408991 7012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:06.416795 7012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:06.416917 7012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:06.416997 7012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:06.417124 7012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:06.417204 7012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:06.426956 7012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38673,127.0.97.125:43235
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.125.keytab
--principal=oryx/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
W20260619 06:28:06.427304 7012 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.126:38673,127.0.97.125:43235'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 06:28:06.428264 7012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:06.429580 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:06.440733 7017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:06.440827 7018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:06.440860 7020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:06.441704 7012 server_base.cc:1034] running on GCE node
I20260619 06:28:06.442463 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:06.443758 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:06.444967 7012 hybrid_clock.cc:648] HybridClock initialized: now 1781850486444902 us; error 48 us; skew 500 ppm
I20260619 06:28:06.445405 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:06 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:06.451210 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.125@KRBTEST.COM (short username oryx)
I20260619 06:28:06.452497 7012 webserver.cc:460] Webserver started at http://127.0.97.125:39009/ using document root <none> and password file <none>
I20260619 06:28:06.453189 7012 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:06.453279 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:06.453523 7012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:06.456176 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/instance:
uuid: "66fd8ba3e4c44e13be8be2fdfe856095"
format_stamp: "Formatted at 2026-06-19 06:28:06 on dist-test-slave-4t1f"
server_key: "5be2776441fc923e3d55e8c33e6bad38"
server_key_iv: "c58ef9a96049e11335d161e4597817e2"
server_key_version: "encryptionkey@0"
I20260619 06:28:06.456846 7012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal/instance:
uuid: "66fd8ba3e4c44e13be8be2fdfe856095"
format_stamp: "Formatted at 2026-06-19 06:28:06 on dist-test-slave-4t1f"
server_key: "5be2776441fc923e3d55e8c33e6bad38"
server_key_iv: "c58ef9a96049e11335d161e4597817e2"
server_key_version: "encryptionkey@0"
I20260619 06:28:06.461455 7012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 06:28:06.464430 7027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:06.465655 7012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:28:06.465816 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
uuid: "66fd8ba3e4c44e13be8be2fdfe856095"
format_stamp: "Formatted at 2026-06-19 06:28:06 on dist-test-slave-4t1f"
server_key: "5be2776441fc923e3d55e8c33e6bad38"
server_key_iv: "c58ef9a96049e11335d161e4597817e2"
server_key_version: "encryptionkey@0"
I20260619 06:28:06.466005 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:06.481350 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:06.485116 7012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:06.485361 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:06.502147 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:43235
I20260619 06:28:06.502177 7078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:43235 every 8 connection(s)
I20260619 06:28:06.503422 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
I20260619 06:28:06.503955 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 7012
I20260619 06:28:06.507454 7079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:06.515386 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:28:06 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
Jun 19 06:28:06 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:06.526805 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.506185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48890 (local address 127.0.97.125:43235)
0619 06:28:06.506979 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.506990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.507029 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:28:06.508600 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.508620 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.508974 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.509327 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.511057 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.512345 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.514068 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.514689 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.514811 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.519506 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.519515 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.519532 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.519571 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.522858 (+ 3287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.523461 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.523473 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.523479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.523622 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.523994 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.524003 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.524005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.524500 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.524787 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.525305 (+ 518us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.525673 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":354,"threads_started":1}
I20260619 06:28:06.533035 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.518344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.0.97.126:38673)
0619 06:28:06.518528 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.518535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.519257 (+ 722us) server_negotiation.cc:407] Connection header received
0619 06:28:06.519462 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.519470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.519549 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.519658 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.520864 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.522045 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.523474 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.523947 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.524070 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.527530 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.527536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.527540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.527590 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.530520 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.531146 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.531152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.531158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.531296 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.531644 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.531651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.531654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.531938 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.532086 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.532489 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.532686 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:28:06.533110 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.518336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:41646)
0619 06:28:06.519112 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:06.519151 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:06.519294 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:06.519689 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:06.519693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:06.520425 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:06.520714 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.520722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.522191 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.522195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.523336 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.523347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.524131 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.524140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.524295 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.524901 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:06.524916 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:06.527358 (+ 2442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:06.530644 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.530656 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.530662 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.531031 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.531402 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.531408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.531411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.531540 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.532059 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:06.532066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:06.532228 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 06:28:06.532404 (+ 176us) client_negotiation.cc:240] Negotiation successful
0619 06:28:06.532688 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":393,"threads_started":1}
I20260619 06:28:06.534479 7079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:28:06 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260619 06:28:06.545341 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.535174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.0.1:48896)
0619 06:28:06.535375 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:06.535397 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:06.535467 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:06.535716 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:06.535718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:06.535989 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:06.536104 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.536110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.536985 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.536988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.537875 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.537885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.538464 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.538468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.538578 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.538983 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:06.538997 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:06.540645 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:06.543424 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.543427 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.543430 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.543805 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.544148 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.544151 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.544154 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.544273 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.544669 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:06.544672 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:06.544773 (+ 101us) client_negotiation.cc:769] Sending connection context
0619 06:28:06.544885 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:28:06.545115 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 06:28:06.545637 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.535387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.0.97.125:43235)
0619 06:28:06.535520 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.535524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.535537 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:28:06.535571 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.535574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.535618 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.535690 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.536206 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.536856 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.537983 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.538343 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.538443 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.540814 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.540819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.540822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.540849 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.543319 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.543923 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.543926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.543928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.544053 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.544355 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.544358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.544360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.544581 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.544680 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.544964 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.545247 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":40}
I20260619 06:28:06.549901 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Bootstrap starting.
I20260619 06:28:06.552647 7079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:06.553653 7079 log.cc:824] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:06.555925 7079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: No bootstrap required, opened a new log
I20260619 06:28:06.562424 7079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } }
I20260619 06:28:06.562696 7079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:06.562813 7079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66fd8ba3e4c44e13be8be2fdfe856095, State: Initialized, Role: FOLLOWER
I20260619 06:28:06.563361 7079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } }
I20260619 06:28:06.564087 7085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } } }
I20260619 06:28:06.564391 7085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:28:06.564917 7079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:28:06.572477 7096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:28:06.572598 7096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:28:06.574944 7012 master_runner.cc:428] Detected that this master 66fd8ba3e4c44e13be8be2fdfe856095 is joining an existing cluster
I20260619 06:28:06.575032 7012 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.125:43235
I20260619 06:28:06.586894 6874 heartbeater.cc:503] Master 127.0.97.126:38673 was elected leader, sending a full tablet report...
I20260619 06:28:06.597484 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.578700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.0.97.125:43235)
0619 06:28:06.578874 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.578879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.579409 (+ 530us) server_negotiation.cc:407] Connection header received
0619 06:28:06.579619 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.579632 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.579706 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.579802 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.581692 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.583128 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.585634 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.586430 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.586534 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.590656 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.590663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.590668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.590701 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.594560 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.595362 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.595367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.595371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.595505 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.595916 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.595921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.595925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.596279 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.596454 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.596902 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.597121 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 06:28:06.598728 7102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.578597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.0.1:48904)
0619 06:28:06.579308 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:06.579328 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:06.579490 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:06.580156 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:06.580163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:06.580957 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:06.581503 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.581518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.583371 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.583378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.585467 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.585485 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.586798 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.586804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.586983 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.589261 (+ 2278us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:06.589285 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:06.590481 (+ 1196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:06.594716 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.594721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.594725 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.595230 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.595641 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.595647 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.595653 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.595812 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.596515 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:06.596519 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:06.596655 (+ 136us) client_negotiation.cc:769] Sending connection context
0619 06:28:06.597977 (+ 1322us) client_negotiation.cc:240] Negotiation successful
0619 06:28:06.598282 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":419,"threads_started":1}
I20260619 06:28:06.601209 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.578388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41662 (local address 127.0.97.126:38673)
0619 06:28:06.578582 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.578589 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.586319 (+ 7730us) server_negotiation.cc:407] Connection header received
0619 06:28:06.586740 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.586746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.586825 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.586896 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.588320 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.589535 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.591387 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.591908 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.592051 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.594195 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.594203 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.594211 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.594252 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.597888 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.598749 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.598757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.598763 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.598909 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.599512 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.599517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.599521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.599884 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.600016 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.600606 (+ 590us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.600836 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:28:06.601439 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.579560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:41662)
0619 06:28:06.586210 (+ 6650us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:06.586238 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:06.586588 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:06.587090 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:06.587097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:06.587823 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:06.588146 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.588160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.589743 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.589748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.591223 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.591233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.592097 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.592103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.592262 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.592783 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:06.592807 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:06.593995 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:06.598028 (+ 4033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.598035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.598041 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.598584 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.599185 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.599193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.599200 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.599382 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.600107 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:06.600114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:06.600316 (+ 202us) client_negotiation.cc:769] Sending connection context
0619 06:28:06.600501 (+ 185us) client_negotiation.cc:240] Negotiation successful
0619 06:28:06.600855 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6514,"thread_start_us":6777,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.124@KRBTEST.COM" created.
Jun 19 06:28:06 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260619 06:28:06.624250 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.607292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.0.1:48918)
0619 06:28:06.608022 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:06.608063 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:06.608211 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:06.608627 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:06.608632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:06.609328 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:06.609531 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.609539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.610942 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.610947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.612490 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.612505 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.613866 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.613874 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.614124 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.614784 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:06.614800 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:06.617030 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:06.620869 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.620884 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.620894 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.621561 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.622007 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.622015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.622022 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.622197 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.622978 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:06.622986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:06.623209 (+ 223us) client_negotiation.cc:769] Sending connection context
0619 06:28:06.623411 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 06:28:06.623776 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":252,"threads_started":1}
I20260619 06:28:06.625177 7082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.607318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48918 (local address 127.0.97.125:43235)
0619 06:28:06.607484 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.607490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.608152 (+ 662us) server_negotiation.cc:407] Connection header received
0619 06:28:06.608376 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.608382 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.608480 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.608581 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.609656 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.610801 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.612661 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.613516 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.613663 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.617224 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.617235 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.617248 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.617281 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.620732 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.621735 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.621740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.621744 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.621885 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.622322 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.622327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.622330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.622747 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.622909 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.624054 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.624400 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:28:06.626997 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.125:43235: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.125"
port: 43235
}
attrs {
promote: true
}
}
dest_uuid: "682d9d818385488cb67aa771b5305d6a"
cas_config_opid_index: -1
I20260619 06:28:06.628155 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } }
I20260619 06:28:06.631330 7109 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 66fd8ba3e4c44e13be8be2fdfe856095 (127.0.97.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } }
I20260619 06:28:06.632282 7054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.124.keytab.
Entry for principal oryx/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.124.keytab.
I20260619 06:28:06.635753 7054 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Refusing update from remote peer 682d9d818385488cb67aa771b5305d6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 06:28:06.636520 7109 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.125:43235
I20260619 06:28:06.636689 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35769
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.124.keytab
--principal=oryx/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:28:06.636902 7109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } }
I20260619 06:28:06.637115 7109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:06.641661 7012 master.cc:537] Master@127.0.97.125:43235 shutting down...
I20260619 06:28:06.645390 7108 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:06.653466 7012 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:06.653842 7012 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:06.654045 7012 tablet_replica.cc:330] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: stopping tablet replica
W20260619 06:28:06.655438 6535 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 06:28:06.661641 7012 master.cc:552] Master@127.0.97.125:43235 shutdown complete.
I20260619 06:28:06.661758 7012 master_runner.cc:305] Clearing existing system tablet
I20260619 06:28:06.662210 7012 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:06.662310 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260619 06:28:06.662905 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a -> Peer 66fd8ba3e4c44e13be8be2fdfe856095 (127.0.97.125:43235): Couldn't send request to peer 66fd8ba3e4c44e13be8be2fdfe856095. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:28:06.664638 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260619 06:28:06.669215 7115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:06.669499 7012 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:06.669775 7012 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260619 06:28:06.669860 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
uuid: "66fd8ba3e4c44e13be8be2fdfe856095"
format_stamp: "Formatted at 2026-06-19 06:28:06 on dist-test-slave-4t1f"
server_key: "5be2776441fc923e3d55e8c33e6bad38"
server_key_iv: "c58ef9a96049e11335d161e4597817e2"
server_key_version: "encryptionkey@0"
I20260619 06:28:06.672916 7012 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:06.677922 7012 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:28:06.678076 7012 log.cc:1196] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 06:28:06.678471 7012 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Deleting consensus metadata
I20260619 06:28:06.678699 7012 master_runner.cc:315] Copying system tablet from 127.0.97.126:38673
I20260619 06:28:06.682508 7012 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.126:38673
I20260619 06:28:06.695844 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.683297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.0.97.126:38673)
0619 06:28:06.683518 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:06.683527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:06.685574 (+ 2047us) server_negotiation.cc:407] Connection header received
0619 06:28:06.685776 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:06.685782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:06.685862 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:06.685958 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:06.686696 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.687397 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.688576 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.689013 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.689183 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.690923 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:06.690934 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:06.690938 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:06.690966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:06.693484 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.694057 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.694079 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.694082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.694182 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:06.694504 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:06.694511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:06.694515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:06.694764 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:06.694900 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:06.695294 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 06:28:06.695496 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:28:06.695899 7121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:06.683162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:41672)
0619 06:28:06.685470 (+ 2308us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:06.685489 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:06.685636 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:06.685981 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:06.685986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:06.686363 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:06.686556 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.686565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.687537 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.687542 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.688446 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:06.688457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:06.689195 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:06.689201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:06.689377 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:06.689999 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:06.690015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:06.690740 (+ 725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:06.693603 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.693610 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.693614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.693940 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.694283 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:06.694288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:06.694291 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:06.694409 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:06.694867 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:06.694872 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:06.695055 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 06:28:06.695202 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 06:28:06.695496 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2162,"thread_start_us":2045,"threads_started":1}
I20260619 06:28:06.696775 6571 tablet_copy_service.cc:140] P 682d9d818385488cb67aa771b5305d6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 66fd8ba3e4c44e13be8be2fdfe856095 ({username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:41672)
I20260619 06:28:06.697062 6571 tablet_copy_service.cc:161] P 682d9d818385488cb67aa771b5305d6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 66fd8ba3e4c44e13be8be2fdfe856095 at {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:41672: session id = 66fd8ba3e4c44e13be8be2fdfe856095-00000000000000000000000000000000
I20260619 06:28:06.701644 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:28:06.703753 7012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:28:06.710624 7012 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: tablet copy: Starting download of 0 data blocks...
I20260619 06:28:06.710973 7012 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: tablet copy: Starting download of 1 WAL segments...
I20260619 06:28:06.713768 7012 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:28:06.718199 6571 tablet_copy_service.cc:342] P 682d9d818385488cb67aa771b5305d6a: Request end of tablet copy session 66fd8ba3e4c44e13be8be2fdfe856095-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:41672
I20260619 06:28:06.718389 6571 tablet_copy_service.cc:433] P 682d9d818385488cb67aa771b5305d6a: ending tablet copy session 66fd8ba3e4c44e13be8be2fdfe856095-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 66fd8ba3e4c44e13be8be2fdfe856095
I20260619 06:28:06.730528 7012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:06.734707 7122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:06.735028 7123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:06.744071 7012 server_base.cc:1034] running on GCE node
W20260619 06:28:06.744781 7126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:06.745235 7012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:06.745872 7012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:06.747079 7012 hybrid_clock.cc:648] HybridClock initialized: now 1781850486747003 us; error 51 us; skew 500 ppm
I20260619 06:28:06.747284 7012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:06 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850486, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:06.749503 7012 init.cc:377] Logged in from keytab as oryx/127.0.97.125@KRBTEST.COM (short username oryx)
I20260619 06:28:06.752272 7012 webserver.cc:460] Webserver started at http://127.0.97.125:45763/ using document root <none> and password file <none>
I20260619 06:28:06.752610 7012 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:06.752696 7012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:06.755685 7012 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:28:06.757633 7132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:06.758035 7012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260619 06:28:06.758210 7012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
uuid: "66fd8ba3e4c44e13be8be2fdfe856095"
format_stamp: "Formatted at 2026-06-19 06:28:06 on dist-test-slave-4t1f"
server_key: "5be2776441fc923e3d55e8c33e6bad38"
server_key_iv: "c58ef9a96049e11335d161e4597817e2"
server_key_version: "encryptionkey@0"
I20260619 06:28:06.758430 7012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:06.780938 7012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:06.784843 7012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:06.813391 7012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:43235
I20260619 06:28:06.813433 7183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:43235 every 8 connection(s)
I20260619 06:28:06.814558 7012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
I20260619 06:28:06.822207 7184 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:28:06.826092 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Bootstrap starting.
I20260619 06:28:06.843339 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:28:06.843820 7184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Bootstrap complete.
I20260619 06:28:06.844791 7184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } }
I20260619 06:28:06.845058 7184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 66fd8ba3e4c44e13be8be2fdfe856095, State: Initialized, Role: LEARNER
I20260619 06:28:06.845319 7184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } }
I20260619 06:28:06.846100 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } }
I20260619 06:28:06.846298 7184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:06.846415 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:28:06.855517 7198 catalog_manager.cc:1259] Loaded cluster ID: 53557a40c9e54368b7ff1a27442b6eb3
I20260619 06:28:06.855598 7198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: loading cluster ID for follower catalog manager: success
I20260619 06:28:06.858973 7198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: acquiring CA information for follower catalog manager: success
I20260619 06:28:06.861047 7198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:28:07.006616 7112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:07.006938 7112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:07.007081 7112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:07.014207 7112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:07.014346 7112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:07.014432 7112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:07.014535 7112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:07.014613 7112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:07.024251 7112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.124.keytab
--principal=oryx/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:35769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:07.025719 7112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:07.027230 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:07.038821 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:07.039942 7206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:07.039966 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:07.040228 7112 server_base.cc:1034] running on GCE node
I20260619 06:28:07.041471 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:07.043103 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:07.044322 7112 hybrid_clock.cc:648] HybridClock initialized: now 1781850487044267 us; error 49 us; skew 500 ppm
I20260619 06:28:07.044651 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:07.047340 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.039925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.0.1:48924)
0619 06:28:07.040074 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.040092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.040222 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.041298 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.041306 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.041347 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:07.041533 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.041543 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.043407 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.043412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.044803 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.044815 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.046560 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.046566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.046792 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.046842 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.046943 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.047075 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260619 06:28:07.047394 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.040298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.0.97.125:43235)
0619 06:28:07.040891 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.040900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.040925 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:28:07.040994 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.041001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.041111 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.041249 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:07.041724 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.043266 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.044943 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.046400 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.046528 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.046735 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.046967 (+ 232us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.047086 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":343,"threads_started":1}
I20260619 06:28:07.047950 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.124@KRBTEST.COM (short username oryx)
I20260619 06:28:07.048429 7159 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260619 06:28:07.048955 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 682d9d818385488cb67aa771b5305d6a. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } }
I20260619 06:28:07.049263 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:28:07.050153 7112 webserver.cc:460] Webserver started at http://127.0.97.124:34783/ using document root <none> and password file <none>
I20260619 06:28:07.050818 7112 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:07.050966 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:07.051257 7112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:07.053454 7007 heartbeater.cc:503] Master 127.0.97.126:38673 was elected leader, sending a full tablet report...
I20260619 06:28:07.054114 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/instance:
uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
format_stamp: "Formatted at 2026-06-19 06:28:07 on dist-test-slave-4t1f"
server_key: "37e8c73f9172963938c25dd43aab6f84"
server_key_iv: "bfee1081942adc15ca4384766f68eee6"
server_key_version: "encryptionkey@0"
I20260619 06:28:07.054890 7112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal/instance:
uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
format_stamp: "Formatted at 2026-06-19 06:28:07 on dist-test-slave-4t1f"
server_key: "37e8c73f9172963938c25dd43aab6f84"
server_key_iv: "bfee1081942adc15ca4384766f68eee6"
server_key_version: "encryptionkey@0"
I20260619 06:28:07.059573 7112 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:07.062667 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:07.063869 7112 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:28:07.064018 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
format_stamp: "Formatted at 2026-06-19 06:28:07 on dist-test-slave-4t1f"
server_key: "37e8c73f9172963938c25dd43aab6f84"
server_key_iv: "bfee1081942adc15ca4384766f68eee6"
server_key_version: "encryptionkey@0"
I20260619 06:28:07.064180 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:07.085357 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:07.089025 7112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:07.089351 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:07.106828 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:35769
I20260619 06:28:07.106858 7265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:35769 every 8 connection(s)
I20260619 06:28:07.108083 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
I20260619 06:28:07.111546 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:07.116906 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 7112
I20260619 06:28:07.118613 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260619 06:28:07.140637 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.121378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.0.97.126:38673)
0619 06:28:07.121591 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.121598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.122287 (+ 689us) server_negotiation.cc:407] Connection header received
0619 06:28:07.122603 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.122609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.122679 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.122777 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.124417 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.125620 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.127610 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.128262 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.128388 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.133481 (+ 5093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.133487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.133493 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.133531 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.137326 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.138182 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.138189 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.138193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.138340 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.138748 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.138754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.138758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.139159 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.139321 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.140102 (+ 781us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.140272 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:28:07.141760 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.121337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:41684)
0619 06:28:07.122149 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.122194 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.122385 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.122851 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.122860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.123739 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.124250 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.124272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.125742 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.125745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.127469 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.127483 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.128482 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.128489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.128663 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.129813 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.129848 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.133155 (+ 3307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.137478 (+ 4323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.137488 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.137500 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.138071 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.138447 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.138455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.138459 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.138643 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.139285 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.139291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.139729 (+ 438us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.140035 (+ 306us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.140522 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":433,"threads_started":1}
I20260619 06:28:07.142007 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.118965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.0.97.124:35769)
0619 06:28:07.119689 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.119705 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.119750 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:28:07.121957 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.121977 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.122411 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.122652 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.124422 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.125997 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.127604 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.128394 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.128553 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.134544 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.134558 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.134576 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.134637 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.138712 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.139423 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.139429 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.139432 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.139587 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.139934 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.139938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.139941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.140307 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.140557 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.141437 (+ 880us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.141597 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":329,"threads_started":1}
I20260619 06:28:07.143256 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260619 06:28:07.154712 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.143889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.0.1:48936)
0619 06:28:07.144061 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.144078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.144172 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.144636 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.144639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.144914 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.145096 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.145109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.146027 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.146031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.147131 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.147142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.147848 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.147852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.147993 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.148438 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.148454 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.150288 (+ 1834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.152934 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.152937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.152940 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.153340 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.153628 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.153631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.153633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.153744 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.154143 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.154146 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.154243 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.154334 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.154513 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 06:28:07.154771 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.143956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.0.97.125:43235)
0619 06:28:07.144136 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.144142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.144160 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:28:07.144291 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.144296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.144365 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.144476 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.145236 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.145902 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.147259 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.147707 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.147837 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.150469 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.150474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.150478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.150505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.152821 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.153428 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.153431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.153434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.153545 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.153832 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.153835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.153837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.154058 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.154159 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.154392 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.154557 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:28:07.155664 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260619 06:28:07.167184 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.156299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.0.97.124:35769)
0619 06:28:07.156451 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.156458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.156535 (+ 77us) server_negotiation.cc:407] Connection header received
0619 06:28:07.156681 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.156686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.156736 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.156822 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.157477 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.158151 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.159282 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.159651 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.159785 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.162309 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.162313 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.162316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.162340 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.164431 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.164983 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.164991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.164995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.165143 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.165499 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.165503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.165507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.165744 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.165874 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.166188 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.166493 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":50}
I20260619 06:28:07.167182 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.156252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.0.1:53984)
0619 06:28:07.156451 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.156470 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.156588 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.156834 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.156838 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.157229 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.157353 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.157360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.158253 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.158261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.159166 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.159177 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.159789 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.159794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.159968 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.160405 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.160421 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.162144 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.164553 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.164560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.164564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.164872 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.165245 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.165249 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.165253 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.165393 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.165844 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.165848 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.165990 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.166116 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.166355 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:28:07.171684 7266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Bootstrap starting.
I20260619 06:28:07.174736 7266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:07.175666 7266 log.cc:824] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:07.178077 7266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: No bootstrap required, opened a new log
I20260619 06:28:07.184985 7266 raft_consensus.cc:348] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } }
I20260619 06:28:07.185324 7266 raft_consensus.cc:374] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:07.185421 7266 raft_consensus.cc:729] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db736c9a0dea433cb55f0d4af19bcfa4, State: Initialized, Role: FOLLOWER
I20260619 06:28:07.185966 7266 consensus_queue.cc:260] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } }
I20260619 06:28:07.187080 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } } }
I20260619 06:28:07.187409 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:28:07.188055 7266 sys_catalog.cc:564] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:28:07.195955 7283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:28:07.196532 7283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:28:07.196550 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 6612
I20260619 06:28:07.201145 7112 master_runner.cc:428] Detected that this master db736c9a0dea433cb55f0d4af19bcfa4 is joining an existing cluster
I20260619 06:28:07.201254 7112 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.124:35769
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.65@KRBTEST.COM
I20260619 06:28:07.221446 7285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.197525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38038 (local address 127.0.97.65:44529)
0619 06:28:07.198188 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.198197 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.198224 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:28:07.198314 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.198322 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.198538 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.198685 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.200972 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.202552 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.204471 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.205385 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.205515 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.212329 (+ 6814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.212346 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.212366 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.212423 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.217139 (+ 4716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.218135 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.218147 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.218153 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.218348 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.218865 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.218873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.218879 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.219386 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.219726 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.220472 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.220812 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":320,"threads_started":1}
I20260619 06:28:07.222143 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.206252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.0.97.126:38673)
0619 06:28:07.206450 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.206457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.206535 (+ 78us) server_negotiation.cc:407] Connection header received
0619 06:28:07.206790 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.206795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.206871 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.206954 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.208532 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.209656 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.211688 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.212392 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.212490 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.215604 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.215611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.215616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.215653 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.218915 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.219696 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.219701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.219705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.219843 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.220316 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.220321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.220324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.220702 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.220824 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.221535 (+ 711us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.221754 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:28:07.223721 6735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:07.224246 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.205844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:41700)
0619 06:28:07.206441 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.206460 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.206639 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.207145 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.207150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.207949 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.208361 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.208374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.209828 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.209833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.211521 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.211537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.212754 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.212762 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.212960 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.213838 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.213861 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.215402 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.219084 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.219089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.219093 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.219555 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.220000 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.220004 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.220008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.220192 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.220951 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.220956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.221191 (+ 235us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.223463 (+ 2272us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.223755 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":625,"threads_started":1}
I20260619 06:28:07.244762 7290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.207552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.0.1:48952)
0619 06:28:07.224466 (+ 16914us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.224485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.224599 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.227595 (+ 2996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.227600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.228092 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.228256 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.228266 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.230067 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.230075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.233178 (+ 3103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.233197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.234673 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.234683 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.235052 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.235789 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.235830 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.237266 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.241512 (+ 4246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.241518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.241523 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.242235 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.242722 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.242728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.242732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.242917 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.243660 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.243666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.243810 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.243972 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.244268 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16814,"mutex_wait_us":8,"thread_start_us":18902,"threads_started":1}
I20260619 06:28:07.245851 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.207018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.0.97.125:43235)
0619 06:28:07.207190 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.207196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.224780 (+ 17584us) server_negotiation.cc:407] Connection header received
0619 06:28:07.224877 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.224886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.224988 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.226317 (+ 1329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.228452 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.229861 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.233379 (+ 3518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.234281 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.234422 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.237494 (+ 3072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.237503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.237510 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.237559 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.241367 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.242370 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.242375 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.242379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.242536 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.243049 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.243055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.243059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.243435 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.243587 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.245249 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.245504 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:28:07.251291 7291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.227864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.0.1:53986)
0619 06:28:07.228633 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.228649 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.228772 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.229312 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.229316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.229891 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.230344 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.230360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.232032 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.232037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.234196 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.234210 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.235570 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.235577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.235739 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.236448 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.236469 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.244154 (+ 7685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.248309 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.248314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.248317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.248755 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.249183 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.249187 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.249191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.249314 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.249978 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.249982 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.250152 (+ 170us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.250319 (+ 167us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.250848 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":515,"spinlock_wait_cycles":19200}
I20260619 06:28:07.251904 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.227008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53986 (local address 127.0.97.124:35769)
0619 06:28:07.227686 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.227694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.228919 (+ 1225us) server_negotiation.cc:407] Connection header received
0619 06:28:07.228971 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.228975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.229081 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.229170 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.230519 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.231823 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.234355 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.235222 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.235343 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.244516 (+ 9173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.244523 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.244528 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.244567 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.248130 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.248875 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.248880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.248883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.249024 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.249420 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.249427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.249431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.249772 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.249905 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.250595 (+ 690us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.251554 (+ 959us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":809,"server-negotiator.queue_time_us":62}
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
I20260619 06:28:07.267962 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.239872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.0.1:54000)
0619 06:28:07.240193 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.240209 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.240332 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.252490 (+ 12158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.252495 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.252970 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.253206 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.253219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.254942 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.254948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.256205 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.256220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.258034 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.258040 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.258210 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.258794 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.258814 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.261401 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.265281 (+ 3880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.265287 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.265290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.265745 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.266191 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.266196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.266202 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.266339 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.266967 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.266972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.267125 (+ 153us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.267290 (+ 165us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.267546 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260619 06:28:07.268606 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.240520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54000 (local address 127.0.97.124:35769)
0619 06:28:07.252112 (+ 11592us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.252118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.252135 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:28:07.252214 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.252219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.252322 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.253356 (+ 1034us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.253564 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.254646 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.256392 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.257882 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.258004 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.261598 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.261604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.261609 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.261647 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.265142 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.265926 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.265932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.265936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.266064 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.266473 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.266478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.266481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.266804 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.267700 (+ 896us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.267816 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.268268 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11493,"thread_start_us":15597,"threads_started":1}
I20260619 06:28:07.269882 6551 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.124:35769: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.124"
port: 35769
}
attrs {
promote: true
}
}
dest_uuid: "682d9d818385488cb67aa771b5305d6a"
cas_config_opid_index: 5
I20260619 06:28:07.271322 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } }
I20260619 06:28:07.272984 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.276703 7108 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.124:35769
I20260619 06:28:07.278079 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 LEARNER]: Refusing update from remote peer 682d9d818385488cb67aa771b5305d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 06:28:07.278682 7112 master.cc:537] Master@127.0.97.124:35769 shutting down...
W20260619 06:28:07.287366 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a -> Peer db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124:35769): Couldn't send request to peer db736c9a0dea433cb55f0d4af19bcfa4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 06:28:07.301563 7297 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260619 06:28:07.311311 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.311724 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:07.315379 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.319729 7112 raft_consensus.cc:2229] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:07.320219 7112 raft_consensus.cc:2258] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:07.320469 7112 tablet_replica.cc:330] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: stopping tablet replica
I20260619 06:28:07.321772 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: NON_VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: true } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.322070 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:28:07.322453 7295 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: attempting to promote NON_VOTER 66fd8ba3e4c44e13be8be2fdfe856095 to VOTER
I20260619 06:28:07.323262 7295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } }
I20260619 06:28:07.326597 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 LEARNER]: Refusing update from remote peer 682d9d818385488cb67aa771b5305d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 06:28:07.326751 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.326262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.0.1:54014)
0619 06:28:07.326482 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.326591 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260619 06:28:07.326941 6535 consensus_peers.cc:487] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a -> Peer db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124:35769): Couldn't send request to peer db736c9a0dea433cb55f0d4af19bcfa4. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.124:35769: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:28:07.327252 7295 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260619 06:28:07.329463 7297 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 66fd8ba3e4c44e13be8be2fdfe856095 (127.0.97.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.330055 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 66fd8ba3e4c44e13be8be2fdfe856095 (127.0.97.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.333844 7294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.334091 7294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:07.333763 7297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.334403 7187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.334633 7187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:28:07.334685 7297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:07.338544 7112 master.cc:552] Master@127.0.97.124:35769 shutdown complete.
I20260619 06:28:07.338630 7112 master_runner.cc:305] Clearing existing system tablet
I20260619 06:28:07.339023 7112 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:07.339123 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:07.341894 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20260619 06:28:07.343570 7301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:07.343951 7112 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:07.344192 7112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 06:28:07.344285 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
format_stamp: "Formatted at 2026-06-19 06:28:07 on dist-test-slave-4t1f"
server_key: "37e8c73f9172963938c25dd43aab6f84"
server_key_iv: "bfee1081942adc15ca4384766f68eee6"
server_key_version: "encryptionkey@0"
I20260619 06:28:07.348899 7112 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:07.355409 7112 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 06:28:07.355558 7112 log.cc:1196] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 06:28:07.356081 7112 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Deleting consensus metadata
I20260619 06:28:07.356315 7112 master_runner.cc:315] Copying system tablet from 127.0.97.126:38673
I20260619 06:28:07.359120 7112 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.126:38673
I20260619 06:28:07.374284 6742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.360191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41710 (local address 127.0.97.126:38673)
0619 06:28:07.360434 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.360440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.360463 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:28:07.360652 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.360658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.360729 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.360791 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.361928 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.363050 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.364748 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.365366 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.365516 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.367754 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.367761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.367766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.367802 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.371055 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.371754 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.371759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.371763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.371926 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.372319 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.372325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.372330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.372657 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.372815 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.373659 (+ 844us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.373917 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260619 06:28:07.374290 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.359849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:41710)
0619 06:28:07.360359 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.360378 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.360505 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.360976 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.360981 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.361515 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.361775 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.361787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.363212 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.363217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.364602 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.364615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.365546 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.365552 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.365750 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.366481 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.366503 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.367556 (+ 1053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.371177 (+ 3621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.371182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.371186 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.371640 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.372041 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.372046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.372050 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.372212 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.372786 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.372794 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.373382 (+ 588us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.373531 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.373834 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":620,"threads_started":1}
I20260619 06:28:07.375180 6571 tablet_copy_service.cc:140] P 682d9d818385488cb67aa771b5305d6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer db736c9a0dea433cb55f0d4af19bcfa4 ({username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:41710)
I20260619 06:28:07.375344 6571 tablet_copy_service.cc:161] P 682d9d818385488cb67aa771b5305d6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer db736c9a0dea433cb55f0d4af19bcfa4 at {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:41710: session id = db736c9a0dea433cb55f0d4af19bcfa4-00000000000000000000000000000000
I20260619 06:28:07.378813 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 6612
I20260619 06:28:07.381059 6571 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 06:28:07.382822 7112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 06:28:07.389568 7112 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: tablet copy: Starting download of 0 data blocks...
I20260619 06:28:07.389825 7112 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: tablet copy: Starting download of 1 WAL segments...
I20260619 06:28:07.392227 7112 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 06:28:07.397053 6571 tablet_copy_service.cc:342] P 682d9d818385488cb67aa771b5305d6a: Request end of tablet copy session db736c9a0dea433cb55f0d4af19bcfa4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.124@KRBTEST.COM'} at 127.0.0.1:41710
I20260619 06:28:07.397223 6571 tablet_copy_service.cc:433] P 682d9d818385488cb67aa771b5305d6a: ending tablet copy session db736c9a0dea433cb55f0d4af19bcfa4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer db736c9a0dea433cb55f0d4af19bcfa4
I20260619 06:28:07.398119 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:44529
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:38673
--webserver_port=36891
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:28:07.406802 7112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:07.409344 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:07.414409 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:07.421351 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:07.422080 7112 server_base.cc:1034] running on GCE node
I20260619 06:28:07.422387 7112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:07.423130 7112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:07.424321 7112 hybrid_clock.cc:648] HybridClock initialized: now 1781850487424266 us; error 33 us; skew 500 ppm
I20260619 06:28:07.424547 7112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:07.426414 7112 init.cc:377] Logged in from keytab as oryx/127.0.97.124@KRBTEST.COM (short username oryx)
I20260619 06:28:07.427558 7112 webserver.cc:460] Webserver started at http://127.0.97.124:45409/ using document root <none> and password file <none>
I20260619 06:28:07.427836 7112 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:07.427953 7112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:07.430198 7112 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:28:07.431739 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:07.432174 7112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 06:28:07.432294 7112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
format_stamp: "Formatted at 2026-06-19 06:28:07 on dist-test-slave-4t1f"
server_key: "37e8c73f9172963938c25dd43aab6f84"
server_key_iv: "bfee1081942adc15ca4384766f68eee6"
server_key_version: "encryptionkey@0"
I20260619 06:28:07.432497 7112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:07.455067 7112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:07.458107 7112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:07.516865 7112 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:35769
I20260619 06:28:07.516901 7370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:35769 every 8 connection(s)
I20260619 06:28:07.517840 7112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
I20260619 06:28:07.525456 7371 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:28:07.527848 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Bootstrap starting.
I20260619 06:28:07.540930 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:28:07.541536 7371 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: Bootstrap complete.
I20260619 06:28:07.542474 7371 raft_consensus.cc:348] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } }
I20260619 06:28:07.542735 7371 raft_consensus.cc:729] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: db736c9a0dea433cb55f0d4af19bcfa4, State: Initialized, Role: LEARNER
I20260619 06:28:07.543006 7371 consensus_queue.cc:260] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } }
I20260619 06:28:07.543781 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.544142 7371 sys_catalog.cc:564] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:07.544312 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: This master's current role is: LEARNER
I20260619 06:28:07.552618 7385 catalog_manager.cc:1259] Loaded cluster ID: 53557a40c9e54368b7ff1a27442b6eb3
I20260619 06:28:07.552724 7385 catalog_manager.cc:1552] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: loading cluster ID for follower catalog manager: success
I20260619 06:28:07.555071 7385 catalog_manager.cc:1574] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: acquiring CA information for follower catalog manager: success
I20260619 06:28:07.556702 7385 catalog_manager.cc:1602] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:28:07.769270 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.763519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.0.97.124:35769)
0619 06:28:07.764253 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.764257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.764273 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:28:07.764330 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.764334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.764397 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.764529 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:07.764926 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.765961 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.767454 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.768265 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.768405 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.768665 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.768817 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.768938 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":471,"threads_started":1}
I20260619 06:28:07.769246 7105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.763356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.0.1:54024)
0619 06:28:07.763567 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.763589 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.763725 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.764560 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.764566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.764594 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:07.764781 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.764790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.766093 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.766101 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.767302 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.767313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.768431 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.768436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.768651 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.768685 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.768797 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.768929 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:28:07.770391 7346 raft_consensus.cc:1203] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 06:28:07.771072 7374 sys_catalog.cc:455] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 682d9d818385488cb67aa771b5305d6a. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:07.771360 7374 sys_catalog.cc:458] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: This master's current role is: LEARNER
W20260619 06:28:07.805269 7308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:07.805524 7308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:07.805629 7308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:07.813174 7308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:07.813282 7308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:07.813441 7308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:07.823575 7308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:44529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=36891
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:07.824864 7308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:07.826633 7308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:07.839965 7391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:07.840011 7392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:07.840147 7394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:07.840644 7308 server_base.cc:1034] running on GCE node
I20260619 06:28:07.841254 7308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:07.842150 7308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:07.843331 7308 hybrid_clock.cc:648] HybridClock initialized: now 1781850487843273 us; error 52 us; skew 500 ppm
I20260619 06:28:07.843614 7308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:07.847146 7308 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260619 06:28:07.848475 7308 webserver.cc:460] Webserver started at http://127.0.97.65:36891/ using document root <none> and password file <none>
I20260619 06:28:07.849056 7308 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:07.849226 7308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:07.854177 7308 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:28:07.856547 7401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:07.857939 7308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:28:07.858084 7308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "22195c66c2644992a4847fa8c008a276"
format_stamp: "Formatted at 2026-06-19 06:28:04 on dist-test-slave-4t1f"
server_key: "66afe5b13ec87c12c830b7e08419d028"
server_key_iv: "d5293639c5abcf35d74d9c2df99c680a"
server_key_version: "encryptionkey@0"
I20260619 06:28:07.858999 7308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:07.870213 7308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:07.874364 7308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:07.874640 7308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:07.875464 7308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:07.876634 7308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:07.876714 7308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:07.876785 7308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:07.876837 7308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:07.909511 7308 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:44529
I20260619 06:28:07.909593 7513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:44529 every 8 connection(s)
I20260619 06:28:07.910912 7308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:07.913980 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 7308
I20260619 06:28:07.914258 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 6746
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.66@KRBTEST.COM
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Jun 19 06:28:07 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850487, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260619 06:28:07.957674 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.914054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:35217 (local address 127.0.97.126:38673)
0619 06:28:07.925908 (+ 11854us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.925917 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.932677 (+ 6760us) server_negotiation.cc:407] Connection header received
0619 06:28:07.936358 (+ 3681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.936366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.936479 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.936571 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.938188 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.939732 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.942599 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.943266 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.943382 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.948974 (+ 5592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.948981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.948985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.949025 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.953254 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.954439 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.954444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.954448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.954599 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.955322 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.955331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.955336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.955725 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.955880 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.956938 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.957193 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11704,"thread_start_us":11607,"threads_started":1}
I20260619 06:28:07.957931 7386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.914929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33155 (local address 127.0.97.124:35769)
0619 06:28:07.915427 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.915437 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.925627 (+ 10190us) server_negotiation.cc:407] Connection header received
0619 06:28:07.933073 (+ 7446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.933082 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.933189 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.933275 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.938249 (+ 4974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.939707 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.942678 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.943301 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.943395 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.949636 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.949640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.949643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.949674 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.953299 (+ 3625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.954439 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.954444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.954448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.954764 (+ 316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.955888 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.955893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.955896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.956271 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.956410 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.957382 (+ 972us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.957616 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
I20260619 06:28:07.959270 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.915564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42852 (local address 127.0.97.66:42923)
0619 06:28:07.931270 (+ 15706us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.931279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.931309 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:28:07.931384 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.931394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.931705 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.934816 (+ 3111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.935115 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.938275 (+ 3160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.940544 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.941402 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.944199 (+ 2797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.950221 (+ 6022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.950238 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.950259 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.950314 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.954730 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.955556 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.955564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.955568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.955736 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.956195 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.956200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.956203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.956618 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.957366 (+ 748us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.958615 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.958802 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15498,"thread_start_us":16305,"threads_started":1}
I20260619 06:28:07.959949 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.925338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34189 (local address 127.0.97.125:43235)
0619 06:28:07.940338 (+ 15000us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:07.940342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:07.940361 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:28:07.940411 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:07.940415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:07.940477 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:07.940592 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:07.941945 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.943100 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.945294 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.946087 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.946209 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.952006 (+ 5797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:07.952014 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:07.952019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:07.952063 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:07.955679 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.956566 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.956571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.956574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.956712 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:07.957450 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:07.957456 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:07.957459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:07.957841 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:07.957987 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:07.959313 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0619 06:28:07.959563 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14905,"thread_start_us":14799,"threads_started":1}
I20260619 06:28:07.960548 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.925140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.97.65:34189)
0619 06:28:07.937858 (+ 12718us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.937970 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.938105 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.940609 (+ 2504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.940613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.941325 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.941751 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.941763 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.943259 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.943265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.945056 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.945090 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.946512 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.946524 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.946779 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.947784 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.947807 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.951725 (+ 3918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.955857 (+ 4132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.955862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.955865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.956410 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.956866 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.956871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.956874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.957309 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.958056 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.958064 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.958202 (+ 138us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.958341 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.958754 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12577,"mutex_wait_us":206,"thread_start_us":7565,"threads_started":1}
I20260619 06:28:07.960659 6869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:07.960641 7519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.914355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.97.65:33155)
0619 06:28:07.925442 (+ 11087us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.925500 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.928852 (+ 3352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.933962 (+ 5110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.933977 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.934950 (+ 973us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.935939 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.935969 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.939931 (+ 3962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.939939 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.942528 (+ 2589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.942541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.943754 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.943761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.944439 (+ 678us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.945043 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.945083 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.949449 (+ 4366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.953436 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.953439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.953448 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.954305 (+ 857us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.955569 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.955574 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.955578 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.955766 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.956490 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.956494 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.956662 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.958155 (+ 1493us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.959042 (+ 887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8945,"mutex_wait_us":779,"spinlock_wait_cycles":20736,"thread_start_us":9682,"threads_started":1}
I20260619 06:28:07.960585 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:07.914355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:35217)
0619 06:28:07.932555 (+ 18200us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:07.932580 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:07.936186 (+ 3606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:07.936857 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:07.936864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:07.937654 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:07.938028 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.938043 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.939913 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.939918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.942414 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:07.942431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:07.943768 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:07.943777 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:07.944129 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:07.944901 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:07.944935 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:07.948719 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:07.953422 (+ 4703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.953431 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.953448 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.954307 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.954937 (+ 630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:07.954940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:07.954945 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:07.955162 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:07.955948 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:07.955956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:07.956454 (+ 498us) client_negotiation.cc:769] Sending connection context
0619 06:28:07.958365 (+ 1911us) client_negotiation.cc:240] Negotiation successful
0619 06:28:07.958860 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18028,"mutex_wait_us":359,"thread_start_us":23012,"threads_started":1}
I20260619 06:28:07.963047 7515 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38673
I20260619 06:28:07.963124 7514 heartbeater.cc:347] Connected to a master server at 127.0.97.124:35769
I20260619 06:28:07.963407 7515 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:07.963404 7514 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:07.963605 7516 heartbeater.cc:347] Connected to a master server at 127.0.97.125:43235
I20260619 06:28:07.963724 7516 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:07.964660 7516 heartbeater.cc:511] Master 127.0.97.125:43235 requested a full tablet report, sending...
I20260619 06:28:07.964706 7514 heartbeater.cc:511] Master 127.0.97.124:35769 requested a full tablet report, sending...
I20260619 06:28:07.964731 7515 heartbeater.cc:511] Master 127.0.97.126:38673 requested a full tablet report, sending...
I20260619 06:28:07.966532 6551 ts_manager.cc:194] Re-registered known tserver with Master: 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529)
I20260619 06:28:07.967133 7149 ts_manager.cc:194] Registered new tserver with Master: 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529)
I20260619 06:28:07.967410 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:35217
I20260619 06:28:07.968919 7336 ts_manager.cc:194] Registered new tserver with Master: 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529)
I20260619 06:28:08.052635 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 6746
I20260619 06:28:08.068785 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:42923
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:38673
--webserver_port=33721
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:28:08.288094 7297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: NON_VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: true } } }
I20260619 06:28:08.288506 7297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:08.290241 7527 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: attempting to promote NON_VOTER db736c9a0dea433cb55f0d4af19bcfa4 to VOTER
I20260619 06:28:08.290769 7527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:08.292435 7159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Refusing update from remote peer 682d9d818385488cb67aa771b5305d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 06:28:08.292565 7346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 LEARNER]: Refusing update from remote peer 682d9d818385488cb67aa771b5305d6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260619 06:28:08.293095 7527 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:28:08.293406 7528 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260619 06:28:08.295610 7528 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.296178 7159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.299256 7528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.299480 7528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:08.299777 7297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.300068 7297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:08.301894 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.302054 7527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.302335 7527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:08.302373 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:28:08.302706 7346 raft_consensus.cc:2932] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.307261 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "682d9d818385488cb67aa771b5305d6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:08.307567 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: This master's current role is: FOLLOWER
W20260619 06:28:08.400305 7526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:08.400625 7526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:08.400796 7526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:08.407917 7526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:08.408059 7526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:08.408250 7526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:08.417800 7526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:42923
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=33721
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:08.419010 7526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:08.420305 7526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:08.433597 7537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:08.433571 7540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:08.433609 7538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:08.434427 7526 server_base.cc:1034] running on GCE node
I20260619 06:28:08.435016 7526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:08.435858 7526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:08.437041 7526 hybrid_clock.cc:648] HybridClock initialized: now 1781850488437004 us; error 39 us; skew 500 ppm
I20260619 06:28:08.437381 7526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:08 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:08.440397 7526 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260619 06:28:08.441789 7526 webserver.cc:460] Webserver started at http://127.0.97.66:33721/ using document root <none> and password file <none>
I20260619 06:28:08.442399 7526 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:08.442543 7526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:08.447824 7526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:28:08.450459 7547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:08.451681 7526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:28:08.451839 7526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "7c6e557d99c841e7b8b2ca2eabac1616"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "b66a1ddfaefccd2b7236fa0ef24e8eb2"
server_key_iv: "1bb7c3ce9da62e717055a91c5447906f"
server_key_version: "encryptionkey@0"
I20260619 06:28:08.452719 7526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:08.471107 7526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:08.475112 7526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:08.475353 7526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:08.476248 7526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:08.477501 7526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:08.477602 7526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:08.477720 7526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:08.477807 7526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:08.515239 7526 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:42923
I20260619 06:28:08.515377 7659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:42923 every 8 connection(s)
I20260619 06:28:08.517032 7526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:08.520495 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 7526
I20260619 06:28:08.520753 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 6879
Jun 19 06:28:08 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.67@KRBTEST.COM
Jun 19 06:28:08 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
Jun 19 06:28:08 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Jun 19 06:28:08 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850488, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:08.581878 7665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:08.522165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35012 (local address 127.0.97.67:39745)
0619 06:28:08.551687 (+ 29522us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:08.551697 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:08.551728 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:28:08.551798 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:08.551805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:08.552102 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:08.552269 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:08.554875 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.556920 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.565256 (+ 8336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.566402 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.566541 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:08.572716 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:08.572729 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:08.572746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:08.572784 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:08.578222 (+ 5438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.579179 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.579190 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.579196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.579364 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.579887 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.579892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.579898 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.580318 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:08.580656 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:08.581233 (+ 577us) server_negotiation.cc:299] Negotiation successful
0619 06:28:08.581405 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29323,"thread_start_us":33532,"threads_started":1}
I20260619 06:28:08.583209 7002 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:08.589260 7664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:08.521668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43919 (local address 127.0.97.124:35769)
0619 06:28:08.522307 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:08.522316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:08.553035 (+ 30719us) server_negotiation.cc:407] Connection header received
0619 06:28:08.556474 (+ 3439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:08.556480 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:08.556553 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:08.556651 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:08.567751 (+ 11100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.569014 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.571941 (+ 2927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.572592 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.572694 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:08.579785 (+ 7091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:08.579794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:08.579798 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:08.579836 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:08.585441 (+ 5605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.586314 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.586322 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.586327 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.586482 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.586976 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.586984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.586989 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.587380 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:08.587532 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:08.588623 (+ 1091us) server_negotiation.cc:299] Negotiation successful
0619 06:28:08.588860 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":258,"threads_started":1}
I20260619 06:28:08.590368 7671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:08.561345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:42813 (local address 127.0.97.126:38673)
0619 06:28:08.567772 (+ 6427us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:08.567778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:08.567795 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:28:08.567848 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:08.567855 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:08.567942 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:08.568066 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:08.570820 (+ 2754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.571886 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.573913 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.574693 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.574839 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:08.580061 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:08.580067 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:08.580072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:08.580118 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:08.585619 (+ 5501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.588241 (+ 2622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.588247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.588250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.588402 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.588940 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.588945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.588949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.589289 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:08.589428 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:08.589788 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 06:28:08.590003 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6298,"thread_start_us":6210,"threads_started":1}
I20260619 06:28:08.591683 7668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:08.557755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.97.66:52397)
0619 06:28:08.558368 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:08.558391 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:08.558506 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:08.567340 (+ 8834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:08.567348 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:08.567953 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:08.568201 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:08.568213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.570770 (+ 2557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.570775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:08.572592 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:08.572605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.573800 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.573804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:08.573960 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:08.574563 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:08.574595 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:08.579243 (+ 4648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:08.583683 (+ 4440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:08.583695 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:08.583716 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:08.584509 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:08.585415 (+ 906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:08.585423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:08.585429 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:08.585695 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:08.587834 (+ 2139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:08.587839 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:08.588170 (+ 331us) client_negotiation.cc:769] Sending connection context
0619 06:28:08.588489 (+ 319us) client_negotiation.cc:240] Negotiation successful
0619 06:28:08.589169 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":42,"spinlock_wait_cycles":58624,"thread_start_us":668,"threads_started":1}
I20260619 06:28:08.591825 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:08.560823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:42813)
0619 06:28:08.561445 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:08.561470 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:08.561576 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:08.568301 (+ 6725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:08.568306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:08.568854 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:08.569188 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:08.569202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.572089 (+ 2887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.572096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:08.573776 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:08.573794 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.574906 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.574914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:08.575234 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:08.576334 (+ 1100us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:08.576369 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:08.579864 (+ 3495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:08.587308 (+ 7444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:08.587312 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:08.587316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:08.588123 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:08.588661 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:08.588665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:08.588669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:08.588831 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:08.589389 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:08.589393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:08.589520 (+ 127us) client_negotiation.cc:769] Sending connection context
0619 06:28:08.589672 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 06:28:08.589947 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":627,"threads_started":1}
I20260619 06:28:08.592564 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:08.520830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.97.66:43919)
0619 06:28:08.552873 (+ 32043us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:08.552936 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:08.556164 (+ 3228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:08.557590 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:08.557606 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:08.558472 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:08.559458 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:08.559488 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.569263 (+ 9775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.569270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:08.571792 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:08.571806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.573024 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.573030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:08.573379 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:08.575091 (+ 1712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:08.575229 (+ 138us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:08.579613 (+ 4384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:08.585607 (+ 5994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:08.585612 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:08.585615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:08.586162 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:08.586653 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:08.586660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:08.586665 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:08.586846 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:08.587649 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:08.587658 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:08.588163 (+ 505us) client_negotiation.cc:769] Sending connection context
0619 06:28:08.589500 (+ 1337us) client_negotiation.cc:240] Negotiation successful
0619 06:28:08.589833 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29289,"mutex_wait_us":159,"spinlock_wait_cycles":16256,"thread_start_us":29051,"threads_started":1}
I20260619 06:28:08.593268 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:08.558254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52397 (local address 127.0.97.125:43235)
0619 06:28:08.566993 (+ 8739us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:08.567003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:08.567024 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:28:08.567081 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:08.567087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:08.567166 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:08.567277 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:08.569397 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.570631 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.572742 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:08.573405 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:08.573521 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:08.579449 (+ 5928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:08.579455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:08.579462 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:08.579524 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:08.583523 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.585002 (+ 1479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.585010 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.585016 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.585209 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:08.586101 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:08.586108 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:08.586114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:08.587457 (+ 1343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:08.591208 (+ 3751us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:08.591359 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 06:28:08.592461 (+ 1102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8606,"thread_start_us":13252,"threads_started":1}
I20260619 06:28:08.593335 7661 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38673
I20260619 06:28:08.593634 7661 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:08.594317 7666 heartbeater.cc:347] Connected to a master server at 127.0.97.125:43235
I20260619 06:28:08.594444 7666 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:08.594559 7661 heartbeater.cc:511] Master 127.0.97.126:38673 requested a full tablet report, sending...
I20260619 06:28:08.594900 7666 heartbeater.cc:511] Master 127.0.97.125:43235 requested a full tablet report, sending...
I20260619 06:28:08.594981 7660 heartbeater.cc:347] Connected to a master server at 127.0.97.124:35769
I20260619 06:28:08.595099 7660 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:08.595562 7660 heartbeater.cc:511] Master 127.0.97.124:35769 requested a full tablet report, sending...
I20260619 06:28:08.596123 6551 ts_manager.cc:194] Re-registered known tserver with Master: 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923)
I20260619 06:28:08.596244 7149 ts_manager.cc:194] Registered new tserver with Master: 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923)
I20260619 06:28:08.597204 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:42813
I20260619 06:28:08.597641 7336 ts_manager.cc:194] Registered new tserver with Master: 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923)
I20260619 06:28:08.675001 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 6879
I20260619 06:28:08.691325 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:39745
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:38673
--webserver_port=45811
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:28:08.970332 7515 heartbeater.cc:503] Master 127.0.97.126:38673 was elected leader, sending a full tablet report...
W20260619 06:28:08.999521 7673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:08.999760 7673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:08.999900 7673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:09.006937 7673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:09.007025 7673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:09.007126 7673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:09.016510 7673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:39745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=45811
--tserver_master_addrs=127.0.97.126:38673,127.0.97.125:43235,127.0.97.124:35769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:09.017763 7673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:09.018996 7673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:09.031844 7678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:09.031790 7679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:09.032353 7681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:09.032835 7673 server_base.cc:1034] running on GCE node
I20260619 06:28:09.033427 7673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:09.034224 7673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:09.035445 7673 hybrid_clock.cc:648] HybridClock initialized: now 1781850489035389 us; error 56 us; skew 500 ppm
I20260619 06:28:09.035727 7673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:09 dist-test-slave-4t1f krb5kdc[6506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850489, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:09.038710 7673 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260619 06:28:09.040012 7673 webserver.cc:460] Webserver started at http://127.0.97.67:45811/ using document root <none> and password file <none>
I20260619 06:28:09.040663 7673 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:09.040786 7673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:09.045635 7673 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:28:09.048336 7688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:09.049607 7673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:28:09.049729 7673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "dc91eac4703c451a88feb1b940ed7833"
format_stamp: "Formatted at 2026-06-19 06:28:05 on dist-test-slave-4t1f"
server_key: "54ee7c44203448cc11bb1d8620fef89d"
server_key_iv: "ac15a5bd806067605c055ace2a08c87f"
server_key_version: "encryptionkey@0"
I20260619 06:28:09.050665 7673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:09.060976 7673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:09.065003 7673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:09.065301 7673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:09.066071 7673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:09.067221 7673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:09.067303 7673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:09.067373 7673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:09.067428 7673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:09.098102 7673 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39745
I20260619 06:28:09.098169 7800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39745 every 8 connection(s)
I20260619 06:28:09.099408 7673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:09.108893 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 7673
Jun 19 06:28:09 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850489, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
Jun 19 06:28:09 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850489, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.124@KRBTEST.COM
Jun 19 06:28:09 dist-test-slave-4t1f krb5kdc[6506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850489, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.125@KRBTEST.COM
I20260619 06:28:09.130534 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.102361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:51211 (local address 127.0.97.126:38673)
0619 06:28:09.102857 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.102864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.104040 (+ 1176us) server_negotiation.cc:407] Connection header received
0619 06:28:09.106890 (+ 2850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.106898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.107005 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.107141 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.110311 (+ 3170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.111305 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.113943 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.114445 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.114599 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.120715 (+ 6116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.120726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.120734 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.120791 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.125665 (+ 4874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.126700 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.126708 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.126714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.126907 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.127424 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.127432 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.127437 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.128039 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.128203 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.129732 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.130011 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":272,"threads_started":1}
I20260619 06:28:09.130683 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.102916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:51211)
0619 06:28:09.103841 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.103899 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.106592 (+ 2693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.107360 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.107371 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.108433 (+ 1062us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:09.109519 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.109549 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.111453 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.111458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.113839 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.113849 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.114787 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.114796 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.115092 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.115841 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:09.115876 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:09.120212 (+ 4336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:09.125855 (+ 5643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:09.125863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:09.125870 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:09.126517 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:09.127058 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:09.127066 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:09.127071 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:09.127276 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:09.128279 (+ 1003us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:09.128287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:09.128519 (+ 232us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.128692 (+ 173us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.129377 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":417,"thread_start_us":347,"threads_started":1}
I20260619 06:28:09.132362 7802 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38673
I20260619 06:28:09.132761 7802 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:09.133882 7802 heartbeater.cc:511] Master 127.0.97.126:38673 requested a full tablet report, sending...
I20260619 06:28:09.134971 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.103388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56263 (local address 127.0.97.124:35769)
0619 06:28:09.105374 (+ 1986us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.105378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.105393 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:28:09.106997 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.107003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.107099 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.107198 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.109715 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.110847 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.114071 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.114794 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.115059 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.119999 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.120006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.120010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.120049 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.130939 (+ 10890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.131700 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.131705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.131708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.131840 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.132297 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.132305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.132310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.132755 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.134002 (+ 1247us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.134172 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.134448 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1894,"thread_start_us":1791,"threads_started":1}
I20260619 06:28:09.135358 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.102891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.97.67:56263)
0619 06:28:09.104858 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.104885 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.106592 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.107359 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.107371 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.108374 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:09.109527 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.109548 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.110995 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.111000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.113929 (+ 2929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.113942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.114994 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.115000 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.115298 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.116069 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:09.116098 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:09.119736 (+ 3638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:09.131083 (+ 11347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:09.131088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:09.131091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:09.131560 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:09.131948 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:09.131956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:09.131962 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:09.132141 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:09.132935 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:09.132944 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:09.133145 (+ 201us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.133345 (+ 200us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.133730 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1767,"mutex_wait_us":496,"spinlock_wait_cycles":41216,"thread_start_us":1443,"threads_started":1}
I20260619 06:28:09.136605 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.104992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54647 (local address 127.0.97.125:43235)
0619 06:28:09.108423 (+ 3431us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.108429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.109023 (+ 594us) server_negotiation.cc:407] Connection header received
0619 06:28:09.109305 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.109311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.109394 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.109546 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.110697 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.111534 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.113780 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.114435 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.114563 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.120664 (+ 6101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.120673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.120679 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.120718 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.124860 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.126068 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.126076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.126082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.126275 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.126902 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.126910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.126916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.127368 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.127530 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.131463 (+ 3933us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.136060 (+ 4597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3325,"thread_start_us":2306,"threads_started":1}
I20260619 06:28:09.136708 6551 ts_manager.cc:194] Re-registered known tserver with Master: dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745)
I20260619 06:28:09.137414 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.104988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.97.67:54647)
0619 06:28:09.108846 (+ 3858us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.108881 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.108994 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.109720 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.109724 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.110256 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:09.110518 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.110529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.111720 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.111725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.113615 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.113629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.114662 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.114669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.115658 (+ 989us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.117422 (+ 1764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:09.117449 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:09.120038 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:09.125104 (+ 5066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:09.125118 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:09.125141 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:09.125859 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:09.126431 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:09.126439 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:09.126445 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:09.126749 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:09.127614 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:09.127627 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:09.128129 (+ 502us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.128474 (+ 345us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.129114 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3714,"mutex_wait_us":751,"thread_start_us":3580,"threads_started":1}
I20260619 06:28:09.138149 7803 heartbeater.cc:347] Connected to a master server at 127.0.97.125:43235
I20260619 06:28:09.138268 7803 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:09.138391 6551 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:51211
I20260619 06:28:09.138690 7803 heartbeater.cc:511] Master 127.0.97.125:43235 requested a full tablet report, sending...
I20260619 06:28:09.139423 7801 heartbeater.cc:347] Connected to a master server at 127.0.97.124:35769
I20260619 06:28:09.139531 7801 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:09.139892 7801 heartbeater.cc:511] Master 127.0.97.124:35769 requested a full tablet report, sending...
I20260619 06:28:09.140094 7149 ts_manager.cc:194] Registered new tserver with Master: dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745)
I20260619 06:28:09.141120 7336 ts_manager.cc:194] Registered new tserver with Master: dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745)
I20260619 06:28:09.157720 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.118688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.0.97.125:43235)
0619 06:28:09.119671 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.119676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.139256 (+ 19580us) server_negotiation.cc:407] Connection header received
0619 06:28:09.139650 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.139658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.139756 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.139851 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.144147 (+ 4296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.144979 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.146495 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.147067 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.147194 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.150624 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.150631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.150636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.150672 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.154225 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.155120 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.155126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.155131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.155300 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.155750 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.155753 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.155755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.156101 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.156720 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.157022 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.157284 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":707,"threads_started":1}
I20260619 06:28:09.160030 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.121816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.97.126:38673)
0619 06:28:09.130852 (+ 9036us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.130857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.143969 (+ 13112us) server_negotiation.cc:407] Connection header received
0619 06:28:09.144031 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.144036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.144120 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.144208 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.145912 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.147505 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.149973 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.150520 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.150644 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.153630 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.153636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.153641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.153698 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.157199 (+ 3501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.157965 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.157970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.157974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.158118 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.158521 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.158526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.158529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.158872 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.159094 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.159434 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.159667 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8939,"thread_start_us":22069,"threads_started":1}
I20260619 06:28:09.162405 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.140390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.0.97.124:35769)
0619 06:28:09.141670 (+ 1280us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.141675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.144027 (+ 2352us) server_negotiation.cc:407] Connection header received
0619 06:28:09.146140 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.146146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.146213 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.146275 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.148966 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.150164 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.151903 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.152604 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.152735 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.155727 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.155733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.155738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.155779 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.159542 (+ 3763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.160272 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.160279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.160282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.160405 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.160758 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.160762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.160765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.161061 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.161280 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.161812 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.162028 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1169}
I20260619 06:28:09.164198 6551 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:28:09.170631 6551 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:09.209323 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.187931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.0.97.66:42923)
0619 06:28:09.200940 (+ 13009us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.200953 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.200989 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:28:09.201118 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.201130 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.201425 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.201716 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:09.202444 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.204683 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.207218 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.208282 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.208408 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.208632 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.208827 (+ 195us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.208996 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12779,"thread_start_us":12659,"threads_started":1}
I20260619 06:28:09.209322 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.187665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42923 (local address 127.0.0.1:37508)
0619 06:28:09.188325 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.188346 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.188464 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.201964 (+ 13500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.201968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.201992 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:09.202279 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.202288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.204804 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.204812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.207038 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.207052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.208446 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.208453 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.208622 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.208669 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.208765 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.208905 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":1,"thread_start_us":397,"threads_started":1}
I20260619 06:28:09.210769 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.188535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39745 (local address 127.0.0.1:35026)
0619 06:28:09.198906 (+ 10371us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.198932 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.199074 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.202886 (+ 3812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.202890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.202918 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:09.203368 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.203380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.205171 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.205175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.207283 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.207295 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.209954 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.209959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.210125 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.210153 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.210247 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.210367 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10208,"mutex_wait_us":46,"thread_start_us":14530,"threads_started":1}
I20260619 06:28:09.211042 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.189292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.0.97.67:39745)
0619 06:28:09.202224 (+ 12932us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.202235 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.202271 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:28:09.202384 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.202394 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.202680 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.202870 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:09.203580 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.205008 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.207429 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.209374 (+ 1945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.209504 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.209691 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.210277 (+ 586us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.210751 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12720,"thread_start_us":12530,"threads_started":1}
I20260619 06:28:09.212332 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.189901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.0.97.65:44529)
0619 06:28:09.201646 (+ 11745us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.201657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.203167 (+ 1510us) server_negotiation.cc:407] Connection header received
0619 06:28:09.203263 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.203270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.203430 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.203655 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:09.204261 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.205508 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.208152 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.211522 (+ 3370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.211616 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.211740 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.212010 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.212126 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11495,"thread_start_us":11368,"threads_started":1}
I20260619 06:28:09.212435 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.188699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44529 (local address 127.0.0.1:57552)
0619 06:28:09.202835 (+ 14136us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.202852 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.202959 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.203658 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.203663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.203688 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:09.204089 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.204099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.205624 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.205635 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.207986 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.208001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.211700 (+ 3699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.211706 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.211861 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.211893 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.211986 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.212091 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14010,"mutex_wait_us":17,"thread_start_us":10463,"threads_started":1}
I20260619 06:28:09.213222 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 678c429d70e041498b81a4221d57b4fe (DEFAULT_TABLE table=test-table [id=13dba1811d19499aa166c77e8669f438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:09.214335 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678c429d70e041498b81a4221d57b4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:09.215358 7449 tablet_service.cc:1467] Processing CreateTablet for tablet 678c429d70e041498b81a4221d57b4fe (DEFAULT_TABLE table=test-table [id=13dba1811d19499aa166c77e8669f438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:09.216575 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678c429d70e041498b81a4221d57b4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:09.224377 7736 tablet_service.cc:1467] Processing CreateTablet for tablet 678c429d70e041498b81a4221d57b4fe (DEFAULT_TABLE table=test-table [id=13dba1811d19499aa166c77e8669f438]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:09.225761 7736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678c429d70e041498b81a4221d57b4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:09.226887 7834 tablet_bootstrap.cc:492] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Bootstrap starting.
I20260619 06:28:09.228564 7835 tablet_bootstrap.cc:492] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Bootstrap starting.
I20260619 06:28:09.231173 7834 tablet_bootstrap.cc:654] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:09.232177 7834 log.cc:824] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:09.232259 7835 tablet_bootstrap.cc:654] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:09.233474 7835 log.cc:824] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:09.235004 7834 tablet_bootstrap.cc:492] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: No bootstrap required, opened a new log
I20260619 06:28:09.235266 7834 ts_tablet_manager.cc:1397] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260619 06:28:09.236219 7835 tablet_bootstrap.cc:492] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: No bootstrap required, opened a new log
I20260619 06:28:09.236881 7835 ts_tablet_manager.cc:1397] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:28:09.241122 7838 tablet_bootstrap.cc:492] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Bootstrap starting.
I20260619 06:28:09.244982 7838 tablet_bootstrap.cc:654] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:09.244793 7834 raft_consensus.cc:348] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.245242 7834 raft_consensus.cc:374] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:09.245349 7834 raft_consensus.cc:729] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6e557d99c841e7b8b2ca2eabac1616, State: Initialized, Role: FOLLOWER
I20260619 06:28:09.245111 7835 raft_consensus.cc:348] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.245514 7835 raft_consensus.cc:374] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:09.245605 7835 raft_consensus.cc:729] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22195c66c2644992a4847fa8c008a276, State: Initialized, Role: FOLLOWER
I20260619 06:28:09.246189 7838 log.cc:824] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:09.246124 7835 consensus_queue.cc:260] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.246095 7834 consensus_queue.cc:260] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.247656 7835 ts_tablet_manager.cc:1428] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260619 06:28:09.248865 7834 ts_tablet_manager.cc:1428] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260619 06:28:09.249380 7661 heartbeater.cc:503] Master 127.0.97.126:38673 was elected leader, sending a full tablet report...
I20260619 06:28:09.263298 7838 tablet_bootstrap.cc:492] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: No bootstrap required, opened a new log
I20260619 06:28:09.263525 7838 ts_tablet_manager.cc:1397] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.008s
I20260619 06:28:09.270113 7838 raft_consensus.cc:348] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.270399 7838 raft_consensus.cc:374] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:09.270455 7838 raft_consensus.cc:729] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc91eac4703c451a88feb1b940ed7833, State: Initialized, Role: FOLLOWER
I20260619 06:28:09.270875 7838 consensus_queue.cc:260] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.271773 7802 heartbeater.cc:503] Master 127.0.97.126:38673 was elected leader, sending a full tablet report...
I20260619 06:28:09.272374 7838 ts_tablet_manager.cc:1428] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
W20260619 06:28:09.314465 7667 tablet.cc:2369] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:28:09.353325 7804 tablet.cc:2369] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:28:09.354632 7840 raft_consensus.cc:482] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:09.354974 7840 raft_consensus.cc:504] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.357059 7840 leader_election.cc:288] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529), dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745)
I20260619 06:28:09.363986 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.357890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:47955 (local address 127.0.97.67:39745)
0619 06:28:09.358114 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.358122 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.358603 (+ 481us) server_negotiation.cc:407] Connection header received
0619 06:28:09.358788 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.358796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.358882 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.358987 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:09.359709 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.360455 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.362285 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.363090 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.363266 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.363360 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.363614 (+ 254us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.363733 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:28:09.364240 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.357378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44777 (local address 127.0.97.65:44529)
0619 06:28:09.357572 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.357579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.359222 (+ 1643us) server_negotiation.cc:407] Connection header received
0619 06:28:09.359361 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.359365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.359422 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.359525 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:09.360044 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.360725 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.362500 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.363335 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.363464 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.363557 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.363868 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.363982 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:28:09.364390 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.357804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39745 (local address 127.0.97.66:47955)
0619 06:28:09.358489 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.358517 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.358647 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.359021 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.359026 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.359054 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:09.359439 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.359460 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.360588 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.360594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.362120 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.362138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.363301 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.363307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.363460 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.363488 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.363594 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.363732 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":24,"spinlock_wait_cycles":25088,"thread_start_us":374,"threads_started":1}
I20260619 06:28:09.364583 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.357824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44529 (local address 127.0.97.66:44777)
0619 06:28:09.359090 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:09.359112 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:09.359241 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:09.359546 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:09.359552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:09.359577 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:09.359894 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.359901 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.360860 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.360866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.362368 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:09.362380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.363525 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.363531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:09.363718 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.363740 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:28:09.363845 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:28:09.363959 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":1013,"threads_started":1}
I20260619 06:28:09.365331 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678c429d70e041498b81a4221d57b4fe" candidate_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc91eac4703c451a88feb1b940ed7833" is_pre_election: true
I20260619 06:28:09.365428 7469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678c429d70e041498b81a4221d57b4fe" candidate_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22195c66c2644992a4847fa8c008a276" is_pre_election: true
I20260619 06:28:09.365875 7469 raft_consensus.cc:2454] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6e557d99c841e7b8b2ca2eabac1616 in term 0.
I20260619 06:28:09.365790 7756 raft_consensus.cc:2454] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c6e557d99c841e7b8b2ca2eabac1616 in term 0.
I20260619 06:28:09.366719 7549 leader_election.cc:302] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22195c66c2644992a4847fa8c008a276, 7c6e557d99c841e7b8b2ca2eabac1616; no voters:
I20260619 06:28:09.367069 7840 raft_consensus.cc:2781] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:09.367208 7840 raft_consensus.cc:482] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:09.367321 7840 raft_consensus.cc:3037] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:09.369412 7840 raft_consensus.cc:504] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.369958 7840 leader_election.cc:288] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [CANDIDATE]: Term 1 election: Requested vote from peers 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529), dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745)
I20260619 06:28:09.370368 7469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678c429d70e041498b81a4221d57b4fe" candidate_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22195c66c2644992a4847fa8c008a276"
I20260619 06:28:09.370374 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678c429d70e041498b81a4221d57b4fe" candidate_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc91eac4703c451a88feb1b940ed7833"
I20260619 06:28:09.370573 7469 raft_consensus.cc:3037] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:09.370580 7756 raft_consensus.cc:3037] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:09.372561 7469 raft_consensus.cc:2454] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6e557d99c841e7b8b2ca2eabac1616 in term 1.
I20260619 06:28:09.372710 7756 raft_consensus.cc:2454] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c6e557d99c841e7b8b2ca2eabac1616 in term 1.
I20260619 06:28:09.372900 7549 leader_election.cc:302] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22195c66c2644992a4847fa8c008a276, 7c6e557d99c841e7b8b2ca2eabac1616; no voters:
I20260619 06:28:09.373207 7840 raft_consensus.cc:2781] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:09.373616 7840 raft_consensus.cc:686] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 LEADER]: Becoming Leader. State: Replica: 7c6e557d99c841e7b8b2ca2eabac1616, State: Running, Role: LEADER
I20260619 06:28:09.374087 7840 consensus_queue.cc:237] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } }
I20260619 06:28:09.378597 6550 catalog_manager.cc:5495] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } health_report { overall_health: UNKNOWN } } }
I20260619 06:28:09.401507 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.396279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.0.97.66:42923)
0619 06:28:09.396456 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.396464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.396484 (+ 20us) server_negotiation.cc:407] Connection header received
0619 06:28:09.396713 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.396719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.396833 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.396997 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:09.397439 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.398203 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.399494 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.399900 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.400024 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.400470 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:09.400964 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:09.401143 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.401190 (+ 47us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.401260 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:28:09.408591 7756 raft_consensus.cc:1261] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Refusing update from remote peer 7c6e557d99c841e7b8b2ca2eabac1616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:09.408602 7469 raft_consensus.cc:1261] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 1 FOLLOWER]: Refusing update from remote peer 7c6e557d99c841e7b8b2ca2eabac1616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:09.409543 7845 consensus_queue.cc:1035] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:09.409874 7840 consensus_queue.cc:1035] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 06:28:09.414891 7517 tablet.cc:2369] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:28:09.421892 7848 mvcc.cc:205] Tried to move back new op lower bound from 7298459604608765952 to 7298459604479262720. Current Snapshot: MvccSnapshot[applied={T|T < 7298459604608765952}]
I20260619 06:28:09.436704 6550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:09.436946 6550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:09.444955 6550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 682d9d818385488cb67aa771b5305d6a: Sending DeleteTablet for 3 replicas of tablet 678c429d70e041498b81a4221d57b4fe
I20260619 06:28:09.446105 7736 tablet_service.cc:1514] Processing DeleteTablet for tablet 678c429d70e041498b81a4221d57b4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:09 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35026
I20260619 06:28:09.446101 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 678c429d70e041498b81a4221d57b4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:09 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37508
I20260619 06:28:09.447512 7449 tablet_service.cc:1514] Processing DeleteTablet for tablet 678c429d70e041498b81a4221d57b4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:09 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57552
I20260619 06:28:09.448083 7448 tablet_service.cc:1514] Processing DeleteTablet for tablet 678c429d70e041498b81a4221d57b4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:09 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57552
I20260619 06:28:09.458282 7858 tablet_replica.cc:330] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: stopping tablet replica
I20260619 06:28:09.458953 7858 raft_consensus.cc:2229] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:28:09.459787 7858 raft_consensus.cc:2258] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:09.462386 7858 ts_tablet_manager.cc:1905] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:09.465214 7859 tablet_replica.cc:330] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: stopping tablet replica
I20260619 06:28:09.465289 7860 tablet_replica.cc:330] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: stopping tablet replica
I20260619 06:28:09.465658 7859 raft_consensus.cc:2229] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:09.465679 7860 raft_consensus.cc:2229] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:09.466169 7859 raft_consensus.cc:2258] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:09.466169 7860 raft_consensus.cc:2258] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:09.468364 7859 ts_tablet_manager.cc:1905] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:09.469838 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 6519
I20260619 06:28:09.470872 6581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:09.471692 7858 ts_tablet_manager.cc:1918] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:09.471988 7858 log.cc:1196] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/678c429d70e041498b81a4221d57b4fe
I20260619 06:28:09.472770 7858 ts_tablet_manager.cc:1939] T 678c429d70e041498b81a4221d57b4fe P 7c6e557d99c841e7b8b2ca2eabac1616: Deleting consensus metadata
I20260619 06:28:09.475517 7859 ts_tablet_manager.cc:1918] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:09.475694 7859 log.cc:1196] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/678c429d70e041498b81a4221d57b4fe
I20260619 06:28:09.476202 7859 ts_tablet_manager.cc:1939] T 678c429d70e041498b81a4221d57b4fe P dc91eac4703c451a88feb1b940ed7833: Deleting consensus metadata
I20260619 06:28:09.481622 7860 ts_tablet_manager.cc:1905] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:09.488726 7860 ts_tablet_manager.cc:1918] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:09.488965 7860 log.cc:1196] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/678c429d70e041498b81a4221d57b4fe
I20260619 06:28:09.489578 7860 ts_tablet_manager.cc:1939] T 678c429d70e041498b81a4221d57b4fe P 22195c66c2644992a4847fa8c008a276: Deleting consensus metadata
I20260619 06:28:09.496403 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.449966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.97.125:43235)
0619 06:28:09.450145 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.450151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.468119 (+ 17968us) server_negotiation.cc:407] Connection header received
0619 06:28:09.479723 (+ 11604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.479730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.479835 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.479946 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.482236 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.483364 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.485349 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.486029 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.486131 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.490074 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.490080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.490085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.490126 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.493546 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.494278 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.494285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.494290 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.494426 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.494866 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.494871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.494875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.495217 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.495463 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.495845 (+ 382us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.496072 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:28:09.513661 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.468668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.0.97.124:35769)
0619 06:28:09.468885 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.468890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.496420 (+ 27530us) server_negotiation.cc:407] Connection header received
0619 06:28:09.496937 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.496942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.497018 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.497109 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.499039 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.500165 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.501989 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.502628 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.502772 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.505982 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.505989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.505994 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.506034 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.509476 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.510457 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.510464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.510468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.510721 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.511319 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.511325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.511328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.511808 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.512090 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.512836 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.513167 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 06:28:09.613814 6537 catalog_manager.cc:4841] TS 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923): tablet 678c429d70e041498b81a4221d57b4fe (table test-table [id=13dba1811d19499aa166c77e8669f438]) successfully deleted
I20260619 06:28:09.614130 6536 catalog_manager.cc:4841] TS 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529): tablet 678c429d70e041498b81a4221d57b4fe (table test-table [id=13dba1811d19499aa166c77e8669f438]) successfully deleted
W20260619 06:28:09.614122 6537 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.66:42923, user_credentials={real_user=slave}} blocked reactor thread for 139060us
I20260619 06:28:09.614522 6537 catalog_manager.cc:4841] TS dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745): tablet 678c429d70e041498b81a4221d57b4fe (table test-table [id=13dba1811d19499aa166c77e8669f438]) successfully deleted
W20260619 06:28:09.614580 6536 catalog_manager.cc:4808] TS 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529): delete failed for tablet 678c429d70e041498b81a4221d57b4fe because the tablet was not found. No further retry: Not found: Tablet not found: 678c429d70e041498b81a4221d57b4fe
I20260619 06:28:09.619908 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 6519
I20260619 06:28:09.664795 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.645671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.0.97.125:43235)
0619 06:28:09.645861 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.645868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.646901 (+ 1033us) server_negotiation.cc:407] Connection header received
0619 06:28:09.647161 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.647167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.647246 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.647317 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.649758 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.650894 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.652921 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.653616 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.653743 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.657802 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.657809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.657814 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.657852 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.661663 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.662649 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.662656 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.662660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.662799 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.663320 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.663325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.663329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.663678 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.663900 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.664210 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.664429 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:28:09.673027 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:09.647205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50864 (local address 127.0.97.124:35769)
0619 06:28:09.647465 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:09.647473 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:09.657976 (+ 10503us) server_negotiation.cc:407] Connection header received
0619 06:28:09.658843 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:09.658852 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:09.658954 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:09.659035 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:09.661467 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.662241 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.663986 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:09.664621 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:09.664734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:09.667362 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:09.667369 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:09.667374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:09.667413 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:09.670615 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.671205 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.671211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.671215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.671349 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:09.671644 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:09.671649 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:09.671652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:09.671996 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:09.672235 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:09.672464 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 06:28:09.672678 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:28:10.429548 7870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.428774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:35925)
0619 06:28:10.429215 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.429333 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":214,"threads_started":1}
W20260619 06:28:10.430030 7661 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:38673 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.433779 7871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.432886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:39641)
0619 06:28:10.433372 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.433576 (+ 204us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":520,"threads_started":1}
W20260619 06:28:10.434155 7802 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:38673 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.448685 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.448067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:44687)
0619 06:28:10.448460 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.448553 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":237,"threads_started":1}
W20260619 06:28:10.449028 7515 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:38673 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.944578 7873 raft_consensus.cc:482] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 682d9d818385488cb67aa771b5305d6a)
I20260619 06:28:10.944803 7874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 682d9d818385488cb67aa771b5305d6a)
I20260619 06:28:10.944900 7873 raft_consensus.cc:504] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:10.945068 7874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:10.952543 7873 leader_election.cc:288] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673), 66fd8ba3e4c44e13be8be2fdfe856095 (127.0.97.125:43235)
I20260619 06:28:10.957927 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.946529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35300)
0619 06:28:10.957438 (+ 10909us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.957626 (+ 188us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10801,"thread_start_us":11020,"threads_started":1}
I20260619 06:28:10.958141 7874 leader_election.cc:288] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673), db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124:35769)
W20260619 06:28:10.958372 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.958535 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.946469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35292)
0619 06:28:10.956945 (+ 10476us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.958169 (+ 1224us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10313,"thread_start_us":10832,"threads_started":1}
W20260619 06:28:10.958767 7322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.965740 7875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.965334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35314)
0619 06:28:10.965532 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.965610 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260619 06:28:10.965943 7322 leader_election.cc:334] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673): Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.967383 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.967011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35326)
0619 06:28:10.967147 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.967224 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:28:10.967608 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.946560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:43235 (local address 127.0.0.1:33568)
0619 06:28:10.957313 (+ 10753us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.957332 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:10.957463 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:10.959035 (+ 1572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:10.959039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:10.959073 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:10.959530 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:10.959546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.962091 (+ 2545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.962099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:10.964852 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:10.964866 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.966722 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.966727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:10.966902 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:10.967045 (+ 143us) client_negotiation.cc:769] Sending connection context
0619 06:28:10.967140 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:28:10.967253 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10614,"mutex_wait_us":62,"thread_start_us":10293,"threads_started":1}
I20260619 06:28:10.967821 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.946592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:35769 (local address 127.0.0.1:50868)
0619 06:28:10.957681 (+ 11089us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.957704 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:10.957845 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:10.959189 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:10.959196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:10.959232 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:10.960221 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:10.960238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.962444 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.962453 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:10.965274 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:10.965293 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.967051 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.967057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:10.967221 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:10.967251 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:28:10.967332 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 06:28:10.967468 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10955,"thread_start_us":10660,"threads_started":1}
I20260619 06:28:10.967944 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.957510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.0.97.125:43235)
0619 06:28:10.958629 (+ 1119us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:10.958634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:10.958652 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:28:10.958707 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:10.958711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:10.958785 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:10.958896 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:10.959995 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.961893 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.965033 (+ 3140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.966344 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.966496 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:10.966606 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:10.967164 (+ 558us) server_negotiation.cc:299] Negotiation successful
0619 06:28:10.967287 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":268,"threads_started":1}
W20260619 06:28:10.968082 7135 leader_election.cc:334] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673): Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.968384 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.947162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50868 (local address 127.0.97.124:35769)
0619 06:28:10.958697 (+ 11535us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:10.958703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:10.958719 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:28:10.958775 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:10.958779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:10.958852 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:10.958953 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:10.960486 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.962168 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.965516 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:10.966738 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:10.966878 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:10.967088 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:10.967464 (+ 376us) server_negotiation.cc:299] Negotiation successful
0619 06:28:10.967600 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":11428,"thread_start_us":11239,"threads_started":1}
I20260619 06:28:10.968497 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" is_pre_election: true
I20260619 06:28:10.968840 7159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db736c9a0dea433cb55f0d4af19bcfa4 in term 1.
I20260619 06:28:10.969018 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" is_pre_election: true
I20260619 06:28:10.969340 7346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66fd8ba3e4c44e13be8be2fdfe856095 in term 1.
I20260619 06:28:10.969307 7320 leader_election.cc:302] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66fd8ba3e4c44e13be8be2fdfe856095, db736c9a0dea433cb55f0d4af19bcfa4; no voters: 682d9d818385488cb67aa771b5305d6a
I20260619 06:28:10.969694 7873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:28:10.969851 7873 raft_consensus.cc:482] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 682d9d818385488cb67aa771b5305d6a)
I20260619 06:28:10.969837 7133 leader_election.cc:302] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66fd8ba3e4c44e13be8be2fdfe856095, db736c9a0dea433cb55f0d4af19bcfa4; no voters: 682d9d818385488cb67aa771b5305d6a
I20260619 06:28:10.969954 7873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:28:10.970170 7874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 06:28:10.970295 7874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 682d9d818385488cb67aa771b5305d6a)
I20260619 06:28:10.970402 7874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:28:10.972112 7873 raft_consensus.cc:504] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:10.972393 7874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:10.972745 7873 leader_election.cc:288] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [CANDIDATE]: Term 2 election: Requested vote from peers 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673), 66fd8ba3e4c44e13be8be2fdfe856095 (127.0.97.125:43235)
I20260619 06:28:10.973035 7874 leader_election.cc:288] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 2 election: Requested vote from peers 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673), db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124:35769)
I20260619 06:28:10.973212 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.972777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35328)
0619 06:28:10.972981 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.973049 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260619 06:28:10.973290 7159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "66fd8ba3e4c44e13be8be2fdfe856095"
I20260619 06:28:10.973567 7159 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate db736c9a0dea433cb55f0d4af19bcfa4 in current term 2: Already voted for candidate 66fd8ba3e4c44e13be8be2fdfe856095 in this term.
I20260619 06:28:10.973580 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
I20260619 06:28:10.973874 7346 raft_consensus.cc:2379] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 66fd8ba3e4c44e13be8be2fdfe856095 in current term 2: Already voted for candidate db736c9a0dea433cb55f0d4af19bcfa4 in this term.
I20260619 06:28:10.974144 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.973770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35354)
0619 06:28:10.973909 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.974001 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 06:28:10.974298 7322 leader_election.cc:334] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673): Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.974434 7322 leader_election.cc:302] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: db736c9a0dea433cb55f0d4af19bcfa4; no voters: 66fd8ba3e4c44e13be8be2fdfe856095, 682d9d818385488cb67aa771b5305d6a
I20260619 06:28:10.974545 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.974143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35340)
0619 06:28:10.974327 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.974414 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:28:10.974714 7873 raft_consensus.cc:2726] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 06:28:10.975286 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:10.975021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35362)
0619 06:28:10.975118 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:10.975182 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260619 06:28:10.975438 7135 leader_election.cc:334] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673): Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:10.975593 7135 leader_election.cc:302] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66fd8ba3e4c44e13be8be2fdfe856095; no voters: 682d9d818385488cb67aa771b5305d6a, db736c9a0dea433cb55f0d4af19bcfa4
I20260619 06:28:10.975857 7874 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260619 06:28:11.431631 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:11.430830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:37397)
0619 06:28:11.431358 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:11.431473 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":263,"threads_started":1}
I20260619 06:28:11.435456 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:11.434705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:33433)
0619 06:28:11.435126 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:11.435218 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":207,"threads_started":1}
I20260619 06:28:11.450273 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:11.449561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:43957)
0619 06:28:11.450055 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:11.450122 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":308,"threads_started":1}
I20260619 06:28:12.433439 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.432540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:41721)
0619 06:28:12.433087 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.433198 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":280,"threads_started":1}
I20260619 06:28:12.436909 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.436196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:42865)
0619 06:28:12.436644 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.436749 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":251,"threads_started":1}
I20260619 06:28:12.451602 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.450953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:51979)
0619 06:28:12.451384 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.451458 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":240,"threads_started":1}
I20260619 06:28:12.477450 7889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:12.477705 7889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:12.478267 7889 leader_election.cc:288] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673), db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124:35769)
I20260619 06:28:12.478755 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" is_pre_election: true
I20260619 06:28:12.478987 7346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66fd8ba3e4c44e13be8be2fdfe856095 in term 2.
I20260619 06:28:12.479270 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.478495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35564)
0619 06:28:12.478987 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.479100 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260619 06:28:12.479434 7133 leader_election.cc:302] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66fd8ba3e4c44e13be8be2fdfe856095, db736c9a0dea433cb55f0d4af19bcfa4; no voters:
I20260619 06:28:12.479744 7889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260619 06:28:12.479887 7889 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:12.480000 7889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:28:12.480731 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.480317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35578)
0619 06:28:12.480483 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.480580 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:28:12.480939 7135 leader_election.cc:334] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673): Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:12.482348 7889 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:12.482843 7889 leader_election.cc:288] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 3 election: Requested vote from peers 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673), db736c9a0dea433cb55f0d4af19bcfa4 (127.0.97.124:35769)
I20260619 06:28:12.483270 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.482985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35586)
0619 06:28:12.483103 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.483166 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 06:28:12.483314 7346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "db736c9a0dea433cb55f0d4af19bcfa4"
I20260619 06:28:12.483541 7346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:28:12.484202 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.483931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35598)
0619 06:28:12.484051 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.484110 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:28:12.484345 7135 leader_election.cc:334] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673): Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
I20260619 06:28:12.485797 7346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66fd8ba3e4c44e13be8be2fdfe856095 in term 3.
I20260619 06:28:12.486167 7133 leader_election.cc:302] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 66fd8ba3e4c44e13be8be2fdfe856095, db736c9a0dea433cb55f0d4af19bcfa4; no voters: 682d9d818385488cb67aa771b5305d6a
I20260619 06:28:12.486415 7889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:28:12.486790 7889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [term 3 LEADER]: Becoming Leader. State: Replica: 66fd8ba3e4c44e13be8be2fdfe856095, State: Running, Role: LEADER
I20260619 06:28:12.487135 7889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } }
I20260619 06:28:12.489277 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66fd8ba3e4c44e13be8be2fdfe856095. Latest consensus state: current_term: 3 leader_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:12.489526 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:12.490093 7894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:12.493566 7894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:12.494382 7894 catalog_manager.cc:1259] Loaded cluster ID: 53557a40c9e54368b7ff1a27442b6eb3
I20260619 06:28:12.494460 7894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:12.495620 7894 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:12.496388 7894 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Loaded TSK: 0
I20260619 06:28:12.497210 7894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:28:12.540911 7148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:28:12.548162 7148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:12.552281 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.551837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35620)
0619 06:28:12.552028 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.552117 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 06:28:12.552357 7346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [term 3 FOLLOWER]: Refusing update from remote peer 66fd8ba3e4c44e13be8be2fdfe856095: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260619 06:28:12.553030 7892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 06:28:12.553289 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:12.552885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35632)
0619 06:28:12.553048 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:12.553156 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 06:28:12.553514 7135 consensus_peers.cc:487] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 -> Peer 682d9d818385488cb67aa771b5305d6a (127.0.97.126:38673): Couldn't send request to peer 682d9d818385488cb67aa771b5305d6a. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 06:28:12.555665 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66fd8ba3e4c44e13be8be2fdfe856095. Latest consensus state: current_term: 3 leader_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:12.555919 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:28:12.557544 7892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:12.557816 7892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:12.557961 7896 sys_catalog.cc:455] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "682d9d818385488cb67aa771b5305d6a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38673 } } peers { permanent_uuid: "66fd8ba3e4c44e13be8be2fdfe856095" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 43235 } attrs { promote: false } } peers { permanent_uuid: "db736c9a0dea433cb55f0d4af19bcfa4" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 35769 } attrs { promote: false } } }
I20260619 06:28:12.558314 7896 sys_catalog.cc:458] T 00000000000000000000000000000000 P db736c9a0dea433cb55f0d4af19bcfa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:28:13.126955 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.126203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35648)
0619 06:28:13.126658 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.126784 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":229,"threads_started":1}
I20260619 06:28:13.128242 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.127798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35656)
0619 06:28:13.127942 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.128063 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:28:13.153904 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.131972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35042 (local address 127.0.97.67:39745)
0619 06:28:13.147570 (+ 15598us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.147590 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.147617 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:28:13.147705 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.147712 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.147802 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.147930 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:13.148434 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.149563 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.151678 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.152870 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.152997 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.153143 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.153461 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.153611 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15451,"thread_start_us":16119,"threads_started":1}
I20260619 06:28:13.153913 7903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.131829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39745 (local address 127.0.0.1:35042)
0619 06:28:13.132025 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.132045 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:13.132163 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:13.147957 (+ 15794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:13.147965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:13.147998 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:13.148299 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.148310 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.149700 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.149707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.151516 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.151533 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.153104 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.153110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.153296 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.153330 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:28:13.153428 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:28:13.153573 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":1,"spinlock_wait_cycles":3968}
I20260619 06:28:13.154356 7910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.132220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:44529 (local address 127.0.0.1:57554)
0619 06:28:13.147909 (+ 15689us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.147928 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:13.148040 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:13.148934 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:13.148937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:13.148955 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:13.149288 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.149300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.150603 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.150608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.152339 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.152349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.153697 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.153703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.153867 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.153915 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 06:28:13.153992 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 06:28:13.154111 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15518,"mutex_wait_us":1,"thread_start_us":15871,"threads_started":1}
I20260619 06:28:13.155082 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.132534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.97.65:44529)
0619 06:28:13.148621 (+ 16087us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.148627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.148646 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:28:13.148743 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.148748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.148817 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.148920 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:13.149427 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.150458 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.152476 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.153449 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.154600 (+ 1151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.154734 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.154774 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.154867 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15955,"thread_start_us":15860,"threads_started":1}
I20260619 06:28:13.155107 7736 tablet_service.cc:1467] Processing CreateTablet for tablet 6aadbd7a9332410384bb79ac058a790f (DEFAULT_TABLE table=test-table [id=cbfb4683279c42f094fd1d094cab9b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:13.155602 7736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aadbd7a9332410384bb79ac058a790f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:13.155823 7449 tablet_service.cc:1467] Processing CreateTablet for tablet 6aadbd7a9332410384bb79ac058a790f (DEFAULT_TABLE table=test-table [id=cbfb4683279c42f094fd1d094cab9b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:13.156286 7449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aadbd7a9332410384bb79ac058a790f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:13.156356 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.132814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42923 (local address 127.0.0.1:37520)
0619 06:28:13.148006 (+ 15192us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.148026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:13.148160 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:13.149439 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:13.149442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:13.149459 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:13.149763 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.149774 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.150968 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.150975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.153106 (+ 2131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.153122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.155767 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.155772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.155890 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.155911 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:28:13.155996 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:28:13.156084 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15074,"mutex_wait_us":1,"thread_start_us":15361,"threads_started":1}
I20260619 06:28:13.157114 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.133240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37520 (local address 127.0.97.66:42923)
0619 06:28:13.142049 (+ 8809us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.142057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.148101 (+ 6044us) server_negotiation.cc:407] Connection header received
0619 06:28:13.148332 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.148339 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.148422 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.148513 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:13.149924 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.150836 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.153236 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.154530 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.154649 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.154787 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.156766 (+ 1979us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.156871 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8632,"thread_start_us":8503,"threads_started":1}
I20260619 06:28:13.157754 7595 tablet_service.cc:1467] Processing CreateTablet for tablet 6aadbd7a9332410384bb79ac058a790f (DEFAULT_TABLE table=test-table [id=cbfb4683279c42f094fd1d094cab9b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:13.158111 7595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6aadbd7a9332410384bb79ac058a790f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:13.165608 7916 tablet_bootstrap.cc:492] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: Bootstrap starting.
I20260619 06:28:13.166563 7914 tablet_bootstrap.cc:492] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: Bootstrap starting.
I20260619 06:28:13.167850 7914 tablet_bootstrap.cc:654] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:13.168116 7916 tablet_bootstrap.cc:654] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:13.169569 7915 tablet_bootstrap.cc:492] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: Bootstrap starting.
I20260619 06:28:13.169757 7914 tablet_bootstrap.cc:492] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: No bootstrap required, opened a new log
I20260619 06:28:13.169878 7914 ts_tablet_manager.cc:1397] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:28:13.170518 7914 raft_consensus.cc:348] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.170686 7914 raft_consensus.cc:374] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:13.170786 7914 raft_consensus.cc:729] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc91eac4703c451a88feb1b940ed7833, State: Initialized, Role: FOLLOWER
I20260619 06:28:13.171020 7915 tablet_bootstrap.cc:654] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:13.171021 7914 consensus_queue.cc:260] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.171808 7914 ts_tablet_manager.cc:1428] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:28:13.173025 7916 tablet_bootstrap.cc:492] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: No bootstrap required, opened a new log
I20260619 06:28:13.173234 7916 ts_tablet_manager.cc:1397] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260619 06:28:13.174100 7916 raft_consensus.cc:348] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.174419 7916 raft_consensus.cc:374] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:13.174533 7916 raft_consensus.cc:729] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22195c66c2644992a4847fa8c008a276, State: Initialized, Role: FOLLOWER
I20260619 06:28:13.174749 7916 consensus_queue.cc:260] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.175567 7916 ts_tablet_manager.cc:1428] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:28:13.176803 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.175821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:59383)
0619 06:28:13.176341 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.176535 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":266,"threads_started":1}
I20260619 06:28:13.177837 7516 heartbeater.cc:503] Master 127.0.97.125:43235 was elected leader, sending a full tablet report...
I20260619 06:28:13.178045 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.177539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:47379)
0619 06:28:13.177730 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.177823 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260619 06:28:13.181245 7803 heartbeater.cc:503] Master 127.0.97.125:43235 was elected leader, sending a full tablet report...
I20260619 06:28:13.182127 7915 tablet_bootstrap.cc:492] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: No bootstrap required, opened a new log
I20260619 06:28:13.182322 7915 ts_tablet_manager.cc:1397] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20260619 06:28:13.183327 7915 raft_consensus.cc:348] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.183634 7915 raft_consensus.cc:374] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:13.183746 7915 raft_consensus.cc:729] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c6e557d99c841e7b8b2ca2eabac1616, State: Initialized, Role: FOLLOWER
I20260619 06:28:13.184015 7915 consensus_queue.cc:260] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.185214 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.172474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:40269)
0619 06:28:13.184828 (+ 12354us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.184948 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12210,"thread_start_us":12103,"threads_started":1}
I20260619 06:28:13.185730 7915 ts_tablet_manager.cc:1428] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260619 06:28:13.186157 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.185141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:52937)
0619 06:28:13.185752 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.185930 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"threads_started":1}
I20260619 06:28:13.186710 7666 heartbeater.cc:503] Master 127.0.97.125:43235 was elected leader, sending a full tablet report...
I20260619 06:28:13.187680 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.187347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:35913)
0619 06:28:13.187507 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.187583 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 06:28:13.188751 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.188444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:43397)
0619 06:28:13.188586 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.188652 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 06:28:13.207660 7922 raft_consensus.cc:482] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:13.207852 7922 raft_consensus.cc:504] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.209497 7922 leader_election.cc:288] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745), 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923)
I20260619 06:28:13.215056 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.209754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47055 (local address 127.0.97.67:39745)
0619 06:28:13.209921 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.209929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.209948 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:28:13.210377 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.210384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.210481 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.210582 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:13.211005 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.211696 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.213475 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.214215 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.214354 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.214463 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.214712 (+ 249us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.214828 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:28:13.215137 7923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.209640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39745 (local address 127.0.97.65:47055)
0619 06:28:13.209803 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.209836 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:13.210011 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:13.210598 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:13.210602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:13.210650 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:13.210878 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.210886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.211819 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.211826 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.213335 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.213346 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.214382 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.214390 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.214561 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.214583 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:28:13.214694 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 06:28:13.214828 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":2,"spinlock_wait_cycles":12672}
I20260619 06:28:13.215387 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.210034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53119 (local address 127.0.97.66:42923)
0619 06:28:13.210247 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.210255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.210585 (+ 330us) server_negotiation.cc:407] Connection header received
0619 06:28:13.210750 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.210755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.210815 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.210898 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:13.211359 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.212002 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.213726 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.214487 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.214636 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.214737 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.214996 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.215115 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:28:13.215638 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.209980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:42923 (local address 127.0.97.65:53119)
0619 06:28:13.210493 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.210510 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:13.210638 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:13.210920 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:13.210924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:13.210945 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:13.211231 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.211238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.212122 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.212129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.213591 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:13.213612 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.214714 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.214722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:13.214857 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.214878 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:28:13.214973 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:28:13.215096 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":39,"thread_start_us":269,"threads_started":1}
I20260619 06:28:13.215776 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aadbd7a9332410384bb79ac058a790f" candidate_uuid: "22195c66c2644992a4847fa8c008a276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc91eac4703c451a88feb1b940ed7833" is_pre_election: true
I20260619 06:28:13.216022 7756 raft_consensus.cc:2454] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22195c66c2644992a4847fa8c008a276 in term 0.
I20260619 06:28:13.216360 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aadbd7a9332410384bb79ac058a790f" candidate_uuid: "22195c66c2644992a4847fa8c008a276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" is_pre_election: true
I20260619 06:28:13.216491 7404 leader_election.cc:302] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22195c66c2644992a4847fa8c008a276, dc91eac4703c451a88feb1b940ed7833; no voters:
I20260619 06:28:13.216694 7615 raft_consensus.cc:2454] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22195c66c2644992a4847fa8c008a276 in term 0.
I20260619 06:28:13.216856 7922 raft_consensus.cc:2781] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:13.216969 7922 raft_consensus.cc:482] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:13.217098 7922 raft_consensus.cc:3037] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:13.219139 7922 raft_consensus.cc:504] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.219641 7922 leader_election.cc:288] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [CANDIDATE]: Term 1 election: Requested vote from peers dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745), 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923)
I20260619 06:28:13.220098 7615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aadbd7a9332410384bb79ac058a790f" candidate_uuid: "22195c66c2644992a4847fa8c008a276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c6e557d99c841e7b8b2ca2eabac1616"
I20260619 06:28:13.220080 7756 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6aadbd7a9332410384bb79ac058a790f" candidate_uuid: "22195c66c2644992a4847fa8c008a276" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc91eac4703c451a88feb1b940ed7833"
I20260619 06:28:13.220297 7615 raft_consensus.cc:3037] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:13.220292 7756 raft_consensus.cc:3037] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:13.223193 7756 raft_consensus.cc:2454] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22195c66c2644992a4847fa8c008a276 in term 1.
I20260619 06:28:13.223193 7615 raft_consensus.cc:2454] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22195c66c2644992a4847fa8c008a276 in term 1.
I20260619 06:28:13.223528 7404 leader_election.cc:302] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22195c66c2644992a4847fa8c008a276, dc91eac4703c451a88feb1b940ed7833; no voters:
I20260619 06:28:13.223786 7922 raft_consensus.cc:2781] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:13.224195 7922 raft_consensus.cc:686] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 1 LEADER]: Becoming Leader. State: Replica: 22195c66c2644992a4847fa8c008a276, State: Running, Role: LEADER
I20260619 06:28:13.224771 7922 consensus_queue.cc:237] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } }
I20260619 06:28:13.225046 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.224659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:46949)
0619 06:28:13.224803 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.224899 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260619 06:28:13.228302 7148 catalog_manager.cc:5495] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22195c66c2644992a4847fa8c008a276 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "22195c66c2644992a4847fa8c008a276" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22195c66c2644992a4847fa8c008a276" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 44529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 } health_report { overall_health: UNKNOWN } } }
I20260619 06:28:13.269498 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.264121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.0.97.65:44529)
0619 06:28:13.264327 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.264336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.264640 (+ 304us) server_negotiation.cc:407] Connection header received
0619 06:28:13.264890 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.264894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.264966 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.265136 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:13.265508 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.266206 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.267485 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.267889 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.268019 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.268424 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:13.268923 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:13.269111 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.269176 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.269250 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:28:13.273260 7756 raft_consensus.cc:1261] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Refusing update from remote peer 22195c66c2644992a4847fa8c008a276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:13.273531 7615 raft_consensus.cc:1261] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 FOLLOWER]: Refusing update from remote peer 22195c66c2644992a4847fa8c008a276: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:13.273945 7927 consensus_queue.cc:1035] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc91eac4703c451a88feb1b940ed7833" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:13.274322 7922 consensus_queue.cc:1035] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c6e557d99c841e7b8b2ca2eabac1616" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 42923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:13.276679 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.276213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:45309)
0619 06:28:13.276357 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.276485 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260619 06:28:13.277868 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.277215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:49581)
0619 06:28:13.277520 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.277627 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260619 06:28:13.285158 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.284693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.67:46855)
0619 06:28:13.284876 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.284967 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 06:28:13.287278 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.285772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.66:44755)
0619 06:28:13.287043 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.287141 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1167}
I20260619 06:28:13.287303 7926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.286724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.97.65:54195)
0619 06:28:13.287063 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.287144 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260619 06:28:13.287580 7934 mvcc.cc:205] Tried to move back new op lower bound from 7298459620440305664 to 7298459620252430336. Current Snapshot: MvccSnapshot[applied={T|T < 7298459620440305664 or (T in {7298459620440305664})}]
I20260619 06:28:13.293355 7149 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:13.293642 7149 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:13.298885 7149 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66fd8ba3e4c44e13be8be2fdfe856095: Sending DeleteTablet for 3 replicas of tablet 6aadbd7a9332410384bb79ac058a790f
I20260619 06:28:13.299912 7449 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aadbd7a9332410384bb79ac058a790f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:13 UTC) from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:57554
I20260619 06:28:13.299973 7595 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aadbd7a9332410384bb79ac058a790f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:13 UTC) from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:37520
I20260619 06:28:13.299993 7736 tablet_service.cc:1514] Processing DeleteTablet for tablet 6aadbd7a9332410384bb79ac058a790f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:13 UTC) from {username='oryx', principal='oryx/127.0.97.125@KRBTEST.COM'} at 127.0.0.1:35042
I20260619 06:28:13.303049 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 7308
I20260619 06:28:13.310575 7944 tablet_replica.cc:330] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: stopping tablet replica
I20260619 06:28:13.310799 7944 raft_consensus.cc:2229] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:13.311079 7944 raft_consensus.cc:2258] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:13.311858 7942 tablet_replica.cc:330] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: stopping tablet replica
I20260619 06:28:13.312044 7942 raft_consensus.cc:2229] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:28:13.312623 7942 raft_consensus.cc:2258] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:13.313416 7944 ts_tablet_manager.cc:1905] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:13.314191 7942 ts_tablet_manager.cc:1905] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:13.320654 7944 ts_tablet_manager.cc:1918] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:13.320818 7944 log.cc:1196] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/6aadbd7a9332410384bb79ac058a790f
I20260619 06:28:13.321404 7944 ts_tablet_manager.cc:1939] T 6aadbd7a9332410384bb79ac058a790f P dc91eac4703c451a88feb1b940ed7833: Deleting consensus metadata
I20260619 06:28:13.322744 7135 catalog_manager.cc:4841] TS dc91eac4703c451a88feb1b940ed7833 (127.0.97.67:39745): tablet 6aadbd7a9332410384bb79ac058a790f (table test-table [id=cbfb4683279c42f094fd1d094cab9b78]) successfully deleted
I20260619 06:28:13.323379 7942 ts_tablet_manager.cc:1918] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:13.323418 7943 tablet_replica.cc:330] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: stopping tablet replica
I20260619 06:28:13.323529 7942 log.cc:1196] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/6aadbd7a9332410384bb79ac058a790f
I20260619 06:28:13.323614 7943 raft_consensus.cc:2229] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:13.323959 7942 ts_tablet_manager.cc:1939] T 6aadbd7a9332410384bb79ac058a790f P 22195c66c2644992a4847fa8c008a276: Deleting consensus metadata
I20260619 06:28:13.323948 7943 raft_consensus.cc:2258] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:13.325030 7134 catalog_manager.cc:4841] TS 22195c66c2644992a4847fa8c008a276 (127.0.97.65:44529): tablet 6aadbd7a9332410384bb79ac058a790f (table test-table [id=cbfb4683279c42f094fd1d094cab9b78]) successfully deleted
I20260619 06:28:13.325140 7943 ts_tablet_manager.cc:1905] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:13.331427 7943 ts_tablet_manager.cc:1918] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:13.331559 7943 log.cc:1196] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/6aadbd7a9332410384bb79ac058a790f
I20260619 06:28:13.332015 7943 ts_tablet_manager.cc:1939] T 6aadbd7a9332410384bb79ac058a790f P 7c6e557d99c841e7b8b2ca2eabac1616: Deleting consensus metadata
I20260619 06:28:13.333207 7135 catalog_manager.cc:4841] TS 7c6e557d99c841e7b8b2ca2eabac1616 (127.0.97.66:42923): tablet 6aadbd7a9332410384bb79ac058a790f (table test-table [id=cbfb4683279c42f094fd1d094cab9b78]) successfully deleted
I20260619 06:28:13.337797 7911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.304292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.0.97.65:44529)
0619 06:28:13.304508 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.304518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.322667 (+ 18149us) server_negotiation.cc:407] Connection header received
0619 06:28:13.323128 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.323136 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.323249 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.323326 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:13.325301 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.326398 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.328137 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.328519 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.328661 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.331127 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:13.331142 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:13.331161 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:13.331202 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:13.335059 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:13.335715 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:13.335725 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:13.335729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:13.335908 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:13.336232 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:13.336238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:13.336243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:13.336683 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:13.337004 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.337190 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.337411 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:28:13.338523 7509 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:13.436553 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 7308
I20260619 06:28:13.454308 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 7526
I20260619 06:28:13.466674 7913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.455017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.0.97.66:42923)
0619 06:28:13.455198 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.455202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.455360 (+ 158us) server_negotiation.cc:407] Connection header received
0619 06:28:13.455490 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.455494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.455558 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.455645 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:13.456960 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.457707 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.458959 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.459412 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.459519 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.461914 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:13.461927 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:13.461942 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:13.461974 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:13.464487 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:13.465031 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:13.465038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:13.465040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:13.465214 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:13.465502 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:13.465505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:13.465507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:13.465816 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:13.466023 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.466235 (+ 212us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.466378 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 06:28:13.467314 7655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:13.577251 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.576769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35672)
0619 06:28:13.576953 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.577035 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:28:13.578212 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.577873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38673 (local address 127.0.0.1:35674)
0619 06:28:13.578015 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:13.578081 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:38673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260619 06:28:13.582160 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 7526
I20260619 06:28:13.599160 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 7673
I20260619 06:28:13.611332 7909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:13.599835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.0.97.67:39745)
0619 06:28:13.600020 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:13.600024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:13.600040 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:28:13.600172 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:13.600175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:13.600234 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:13.600312 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:13.601632 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.602323 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.603436 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:13.603811 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:13.603898 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:13.606405 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:13.606419 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:13.606433 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:13.606474 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:13.608999 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:13.609649 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:13.609655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:13.609657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:13.609781 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:13.610080 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:13.610083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:13.610085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:13.610386 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:13.610589 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:13.610836 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 06:28:13.610980 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:28:13.611918 7796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:13.704646 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 7673
I20260619 06:28:13.720999 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 7012
I20260619 06:28:13.721963 7179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:13.840891 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 7012
I20260619 06:28:13.861706 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 7112
I20260619 06:28:13.862630 7366 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:13.973129 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 7112
2026-06-19T06:28:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11171 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:28:14 dist-test-slave-4t1f krb5kdc[7953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:28:14 dist-test-slave-4t1f krb5kdc[7953](info): set up 2 sockets
Jun 19 06:28:14 dist-test-slave-4t1f krb5kdc[7953](info): commencing operation
krb5kdc: starting...
W20260619 06:28:15.070020 389 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:28:15 dist-test-slave-4t1f krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:28:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab.
Entry for principal oryx/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab.
I20260619 06:28:15.272275 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:32859
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:32859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:15.586086 7966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:15.586366 7966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:15.586539 7966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:15.593487 7966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:15.593575 7966 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 06:28:15.593683 7966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:15.593783 7966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:15.593894 7966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:15.593997 7966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:15.603353 7966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:32859
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.126.keytab
--principal=oryx/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:32859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:15.604530 7966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:15.605847 7966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:15.617187 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:15.617252 7972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:15.617358 7971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:15.618161 7966 server_base.cc:1034] running on GCE node
I20260619 06:28:15.618788 7966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:15.619956 7966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:15.621202 7966 hybrid_clock.cc:648] HybridClock initialized: now 1781850495621144 us; error 37 us; skew 500 ppm
I20260619 06:28:15.621464 7966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:15 dist-test-slave-4t1f krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:15.624348 7966 init.cc:377] Logged in from keytab as oryx/127.0.97.126@KRBTEST.COM (short username oryx)
I20260619 06:28:15.625734 7966 webserver.cc:460] Webserver started at http://127.0.97.126:37711/ using document root <none> and password file <none>
I20260619 06:28:15.626372 7966 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:15.626479 7966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:15.626789 7966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:15.629575 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a"
format_stamp: "Formatted at 2026-06-19 06:28:15 on dist-test-slave-4t1f"
server_key: "94916429c2dae74375c37a79ed820d1c"
server_key_iv: "0096f49a7dd5acf2659548e47e0887e0"
server_key_version: "encryptionkey@0"
I20260619 06:28:15.630355 7966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a"
format_stamp: "Formatted at 2026-06-19 06:28:15 on dist-test-slave-4t1f"
server_key: "94916429c2dae74375c37a79ed820d1c"
server_key_iv: "0096f49a7dd5acf2659548e47e0887e0"
server_key_version: "encryptionkey@0"
I20260619 06:28:15.635003 7966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:15.638229 7981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:15.639439 7966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:28:15.639618 7966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a"
format_stamp: "Formatted at 2026-06-19 06:28:15 on dist-test-slave-4t1f"
server_key: "94916429c2dae74375c37a79ed820d1c"
server_key_iv: "0096f49a7dd5acf2659548e47e0887e0"
server_key_version: "encryptionkey@0"
I20260619 06:28:15.639815 7966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:15.653720 7966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:15.657773 7966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:15.658025 7966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:15.677857 7966 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:32859
I20260619 06:28:15.677879 8042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:32859 every 8 connection(s)
I20260619 06:28:15.679116 7966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:15.680519 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 7966
I20260619 06:28:15.680747 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:28:15.681480 389 external_mini_cluster.cc:1428] Setting key bebb4e03e8f0cd695fe95053c7a82736
I20260619 06:28:15.683133 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:28:15 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:15.695617 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Bootstrap starting.
I20260619 06:28:15.699440 8043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:15.700793 8043 log.cc:824] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:15.703589 8043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: No bootstrap required, opened a new log
I20260619 06:28:15.704171 8046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:15.683194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.0.97.126:32859)
0619 06:28:15.683966 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:15.683979 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:15.684027 (+ 48us) server_negotiation.cc:407] Connection header received
0619 06:28:15.686077 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:15.686103 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:15.686594 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:15.687013 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:15.688172 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:15.690217 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:15.691403 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:15.692419 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:15.692609 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:15.696204 (+ 3595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:15.696213 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:15.696228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:15.696282 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:15.700120 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:15.701029 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:15.701040 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:15.701048 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:15.701264 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:15.701641 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:15.701646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:15.701650 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:15.702129 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:15.702416 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:15.702820 (+ 404us) server_negotiation.cc:299] Negotiation successful
0619 06:28:15.703214 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":345,"threads_started":1}
I20260619 06:28:15.711867 8043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 32859 } }
I20260619 06:28:15.712217 8043 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:15.712320 8043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c91da951f8b44c4bf0c5edd1fcdc93a, State: Initialized, Role: FOLLOWER
I20260619 06:28:15.712815 8043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 32859 } }
I20260619 06:28:15.713055 8043 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:15.713232 8043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:15.713408 8043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:15.715476 8043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 32859 } }
I20260619 06:28:15.715974 8043 leader_election.cc:302] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c91da951f8b44c4bf0c5edd1fcdc93a; no voters:
I20260619 06:28:15.716671 8043 leader_election.cc:288] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:28:15.716766 8048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:15.717448 8048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [term 1 LEADER]: Becoming Leader. State: Replica: 2c91da951f8b44c4bf0c5edd1fcdc93a, State: Running, Role: LEADER
I20260619 06:28:15.718022 8048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 32859 } }
I20260619 06:28:15.718816 8043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:15.720705 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 32859 } } }
I20260619 06:28:15.721037 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:15.722612 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c91da951f8b44c4bf0c5edd1fcdc93a. Latest consensus state: current_term: 1 leader_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c91da951f8b44c4bf0c5edd1fcdc93a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 32859 } } }
I20260619 06:28:15.722859 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:15.730026 8055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:15.735049 8055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:15.743476 8055 catalog_manager.cc:1347] Generated new cluster ID: 69bbbc3e76484e8d96ba8b4c1e4db6d6
I20260619 06:28:15.743624 8055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:15.755257 8055 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:28:15.756388 8055 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:15.769320 8055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Generated new TSK 0
I20260619 06:28:15.770099 8055 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260619 06:28:15.836374 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:32859
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:16.152941 8069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:16.153280 8069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:16.153452 8069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:16.160550 8069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:16.160684 8069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:16.160845 8069 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:28:16.160986 8069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:16.170253 8069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:32859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:16.171398 8069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:16.172731 8069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:16.185364 8075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:16.185420 8074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:16.185388 8077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:16.186369 8069 server_base.cc:1034] running on GCE node
I20260619 06:28:16.186936 8069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:16.187801 8069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:16.189033 8069 hybrid_clock.cc:648] HybridClock initialized: now 1781850496188964 us; error 46 us; skew 500 ppm
I20260619 06:28:16.189357 8069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:16 dist-test-slave-4t1f krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:16.192313 8069 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260619 06:28:16.193722 8069 webserver.cc:460] Webserver started at http://127.0.97.65:43453/ using document root <none> and password file <none>
I20260619 06:28:16.194371 8069 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:16.194499 8069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:16.194798 8069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:16.197489 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "d92372700c0f4e13901a604976d34df5"
format_stamp: "Formatted at 2026-06-19 06:28:16 on dist-test-slave-4t1f"
server_key: "15579dae191a24618ba8b86f383c9df5"
server_key_iv: "5fb6b2b71f5f3b5f124628e185ea4a59"
server_key_version: "encryptionkey@0"
I20260619 06:28:16.198212 8069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "d92372700c0f4e13901a604976d34df5"
format_stamp: "Formatted at 2026-06-19 06:28:16 on dist-test-slave-4t1f"
server_key: "15579dae191a24618ba8b86f383c9df5"
server_key_iv: "5fb6b2b71f5f3b5f124628e185ea4a59"
server_key_version: "encryptionkey@0"
I20260619 06:28:16.202777 8069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:16.206012 8084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:16.207310 8069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:28:16.207487 8069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "d92372700c0f4e13901a604976d34df5"
format_stamp: "Formatted at 2026-06-19 06:28:16 on dist-test-slave-4t1f"
server_key: "15579dae191a24618ba8b86f383c9df5"
server_key_iv: "5fb6b2b71f5f3b5f124628e185ea4a59"
server_key_version: "encryptionkey@0"
I20260619 06:28:16.207695 8069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:16.227788 8069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:16.231722 8069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:16.231974 8069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:16.235241 8069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:16.235423 8069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 06:28:16.235620 8069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:16.236012 8069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:28:16 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:16.256668 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.235148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56927 (local address 127.0.97.126:32859)
0619 06:28:16.236125 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:16.236132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:16.237225 (+ 1093us) server_negotiation.cc:407] Connection header received
0619 06:28:16.239042 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:16.239047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:16.239126 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:16.239269 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:16.240673 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.241464 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.243953 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.244588 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.244689 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.249913 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:16.249921 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:16.249935 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:16.249978 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:16.252995 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.254074 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.254077 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.254080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.254183 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.254668 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.254676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.254682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.255049 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:16.255164 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:16.256220 (+ 1056us) server_negotiation.cc:299] Negotiation successful
0619 06:28:16.256418 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":752,"threads_started":1}
I20260619 06:28:16.258013 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.235911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:32859 (local address 127.0.97.65:56927)
0619 06:28:16.236753 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:16.236804 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:16.238780 (+ 1976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:16.239476 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:16.239483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:16.239938 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:16.240500 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.240516 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.241602 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.241612 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.243754 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.243768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.245094 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.245104 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.245431 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.246190 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:16.246238 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:16.249653 (+ 3415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:16.253208 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.253220 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.253240 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.253949 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.254294 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.254299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.254303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.254547 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.255199 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:16.255210 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:16.255683 (+ 473us) client_negotiation.cc:769] Sending connection context
0619 06:28:16.256037 (+ 354us) client_negotiation.cc:240] Negotiation successful
0619 06:28:16.256632 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":398,"threads_started":1}
I20260619 06:28:16.282850 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.267404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47939 (local address 127.0.97.126:32859)
0619 06:28:16.267578 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:16.267584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:16.268484 (+ 900us) server_negotiation.cc:407] Connection header received
0619 06:28:16.268659 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:16.268663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:16.268752 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:16.268883 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:16.270485 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.271480 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.273468 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.273902 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.274030 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.276756 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:16.276760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:16.276764 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:16.276808 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:16.279994 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.280811 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.280818 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.280821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.280973 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.281502 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.281507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.281510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.281754 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:16.281906 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:16.282302 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 06:28:16.282509 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:28:16.283012 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.267458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:32859 (local address 127.0.97.65:47939)
0619 06:28:16.268363 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:16.268388 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:16.268546 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:16.269157 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:16.269164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:16.269863 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:16.270312 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.270327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.271653 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.271661 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.273308 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.273326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.274105 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.274111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.274456 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.275223 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:16.275251 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:16.276595 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:16.280120 (+ 3525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.280126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.280130 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.280707 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.281148 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.281155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.281160 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.281340 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.281873 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:16.281878 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:16.282054 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 06:28:16.282198 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 06:28:16.282534 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":520,"threads_started":1}
I20260619 06:28:16.290622 8069 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:34261
I20260619 06:28:16.290661 8204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:34261 every 8 connection(s)
I20260619 06:28:16.291790 8069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:16.293730 8205 heartbeater.cc:347] Connected to a master server at 127.0.97.126:32859
I20260619 06:28:16.293963 8205 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:16.294560 8205 heartbeater.cc:511] Master 127.0.97.126:32859 requested a full tablet report, sending...
I20260619 06:28:16.295823 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 8069
I20260619 06:28:16.296046 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:28:16.296684 7998 ts_manager.cc:194] Registered new tserver with Master: d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261)
I20260619 06:28:16.296852 389 external_mini_cluster.cc:1428] Setting key 3f7db78433300e4ba18292451216b7df
I20260619 06:28:16.298837 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:56927
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab.
I20260619 06:28:16.340257 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:32859
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:16.645713 8209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:16.646001 8209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:16.646179 8209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:16.653015 8209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:16.653196 8209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:16.653348 8209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:28:16.653489 8209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:16.662884 8209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:32859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:16.664055 8209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:16.665360 8209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:16.678156 8214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:16.678869 8209 server_base.cc:1034] running on GCE node
W20260619 06:28:16.681640 8215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:16.681861 8217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:16.682525 8209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:16.683387 8209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:16.684607 8209 hybrid_clock.cc:648] HybridClock initialized: now 1781850496684530 us; error 57 us; skew 500 ppm
I20260619 06:28:16.684875 8209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:16 dist-test-slave-4t1f krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:16.687832 8209 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260619 06:28:16.689271 8209 webserver.cc:460] Webserver started at http://127.0.97.66:36131/ using document root <none> and password file <none>
I20260619 06:28:16.689932 8209 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:16.690011 8209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:16.690315 8209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:16.692998 8209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "326c5bf6dee74c8b89088a5b749c7bba"
format_stamp: "Formatted at 2026-06-19 06:28:16 on dist-test-slave-4t1f"
server_key: "54b7f3aeab410a6b1b89f9809d7749eb"
server_key_iv: "87ef31e55749aea34abf29b5542d97f4"
server_key_version: "encryptionkey@0"
I20260619 06:28:16.693775 8209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "326c5bf6dee74c8b89088a5b749c7bba"
format_stamp: "Formatted at 2026-06-19 06:28:16 on dist-test-slave-4t1f"
server_key: "54b7f3aeab410a6b1b89f9809d7749eb"
server_key_iv: "87ef31e55749aea34abf29b5542d97f4"
server_key_version: "encryptionkey@0"
I20260619 06:28:16.698307 8209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 06:28:16.701496 8224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:16.703037 8209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:28:16.703225 8209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "326c5bf6dee74c8b89088a5b749c7bba"
format_stamp: "Formatted at 2026-06-19 06:28:16 on dist-test-slave-4t1f"
server_key: "54b7f3aeab410a6b1b89f9809d7749eb"
server_key_iv: "87ef31e55749aea34abf29b5542d97f4"
server_key_version: "encryptionkey@0"
I20260619 06:28:16.703466 8209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:16.720439 8209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:16.724535 8209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:16.724798 8209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:16.727233 8209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:16.727304 8209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:16.727483 8209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:16.727581 8209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:28:16 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850496, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:16.751081 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.727826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:48747 (local address 127.0.97.126:32859)
0619 06:28:16.728026 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:16.728032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:16.729367 (+ 1335us) server_negotiation.cc:407] Connection header received
0619 06:28:16.732414 (+ 3047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:16.732421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:16.732535 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:16.732669 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:16.734757 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.735759 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.738119 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.738628 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.738760 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.744730 (+ 5970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:16.744739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:16.744745 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:16.744788 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:16.748073 (+ 3285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.748744 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.748751 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.748756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.748911 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.749508 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.749514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.749520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.749812 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:16.749955 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:16.750558 (+ 603us) server_negotiation.cc:299] Negotiation successful
0619 06:28:16.750751 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:28:16.752036 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.728367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:32859 (local address 127.0.97.66:48747)
0619 06:28:16.729201 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:16.729254 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:16.732002 (+ 2748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:16.732899 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:16.732906 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:16.733594 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:16.734524 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.734553 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.735928 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.735942 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.737969 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.737980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.738906 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.738917 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.739282 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.740111 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:16.740149 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:16.744456 (+ 4307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:16.748205 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.748213 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.748226 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.748627 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.749048 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.749056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.749059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.749323 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.749933 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:16.749940 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:16.750236 (+ 296us) client_negotiation.cc:769] Sending connection context
0619 06:28:16.750458 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 06:28:16.750914 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":327,"threads_started":1}
I20260619 06:28:16.769806 8209 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39293
I20260619 06:28:16.769830 8342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39293 every 8 connection(s)
I20260619 06:28:16.771351 8209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:16.773665 8344 heartbeater.cc:347] Connected to a master server at 127.0.97.126:32859
I20260619 06:28:16.773883 8344 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:16.774632 8344 heartbeater.cc:511] Master 127.0.97.126:32859 requested a full tablet report, sending...
I20260619 06:28:16.776003 7998 ts_manager.cc:194] Registered new tserver with Master: 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293)
I20260619 06:28:16.777108 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:48747
I20260619 06:28:16.778035 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 8209
I20260619 06:28:16.778221 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:28:16.778736 389 external_mini_cluster.cc:1428] Setting key 7e9dd984816b204131a3d3aab75d63c1
I20260619 06:28:16.783954 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.770852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:37665 (local address 127.0.97.126:32859)
0619 06:28:16.770993 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:16.770997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:16.771595 (+ 598us) server_negotiation.cc:407] Connection header received
0619 06:28:16.771785 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:16.771789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:16.771854 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:16.771969 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:16.773168 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.774261 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.775762 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.776270 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.776416 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.778298 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:16.778303 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:16.778306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:16.778332 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:16.781387 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.782018 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.782038 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.782042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.782162 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:16.782532 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:16.782539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:16.782542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:16.782836 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:16.782994 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:16.783378 (+ 384us) server_negotiation.cc:299] Negotiation successful
0619 06:28:16.783572 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":46}
I20260619 06:28:16.783987 8343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:16.770906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:32859 (local address 127.0.97.66:37665)
0619 06:28:16.771501 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:16.771516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:16.771659 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:16.771998 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:16.772002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:16.772652 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:16.773010 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.773022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.774385 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.774388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.775632 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:16.775647 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:16.776442 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:16.776448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:16.776600 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:16.777171 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:16.777195 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:16.778077 (+ 882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:16.781540 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.781546 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.781550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.781903 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.782288 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:16.782293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:16.782297 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:16.782422 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:16.782953 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:16.782957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:16.783123 (+ 166us) client_negotiation.cc:769] Sending connection context
0619 06:28:16.783281 (+ 158us) client_negotiation.cc:240] Negotiation successful
0619 06:28:16.783572 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":315,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260619 06:28:16.822126 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:32859
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:36765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:17.131480 8348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:17.131759 8348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:17.131928 8348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:17.138847 8348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:17.138983 8348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:17.139135 8348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 06:28:17.139287 8348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:17.149734 8348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.126:32859
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:17.151235 8348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:17.152872 8348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:17.165910 8354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:17.166095 8353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:17.166071 8356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:17.166937 8348 server_base.cc:1034] running on GCE node
I20260619 06:28:17.167428 8348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:17.168151 8348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:17.169348 8348 hybrid_clock.cc:648] HybridClock initialized: now 1781850497169301 us; error 29 us; skew 500 ppm
I20260619 06:28:17.169624 8348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:17 dist-test-slave-4t1f krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850497, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:17.172540 8348 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260619 06:28:17.173848 8348 webserver.cc:460] Webserver started at http://127.0.97.67:45575/ using document root <none> and password file <none>
I20260619 06:28:17.174496 8348 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:17.174607 8348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:17.174916 8348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:17.177676 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8"
format_stamp: "Formatted at 2026-06-19 06:28:17 on dist-test-slave-4t1f"
server_key: "369043da5505a6ec8b215fe12ac28ad6"
server_key_iv: "aadc7eecab73995a074c0301df1797a5"
server_key_version: "encryptionkey@0"
I20260619 06:28:17.178368 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8"
format_stamp: "Formatted at 2026-06-19 06:28:17 on dist-test-slave-4t1f"
server_key: "369043da5505a6ec8b215fe12ac28ad6"
server_key_iv: "aadc7eecab73995a074c0301df1797a5"
server_key_version: "encryptionkey@0"
I20260619 06:28:17.182958 8348 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:17.186072 8363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:17.187259 8348 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:28:17.187422 8348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8"
format_stamp: "Formatted at 2026-06-19 06:28:17 on dist-test-slave-4t1f"
server_key: "369043da5505a6ec8b215fe12ac28ad6"
server_key_iv: "aadc7eecab73995a074c0301df1797a5"
server_key_version: "encryptionkey@0"
I20260619 06:28:17.187620 8348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:17.200798 8348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:17.204290 8348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:17.204545 8348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:17.207033 8348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:17.207127 8348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:17.207247 8348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:17.207320 8348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:28:17 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850497, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:17.231205 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.207572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35209 (local address 127.0.97.126:32859)
0619 06:28:17.207758 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.207765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.209115 (+ 1350us) server_negotiation.cc:407] Connection header received
0619 06:28:17.211157 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.211163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.211248 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.211357 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:17.213808 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.214773 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.217768 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.218363 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.218501 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.223789 (+ 5288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:17.223796 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:17.223802 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:17.223846 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:17.227048 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.228169 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.228176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.228182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.228352 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.228989 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.228996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.229000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.229462 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:17.229626 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.230789 (+ 1163us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.230946 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:28:17.233091 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.208218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:32859 (local address 127.0.97.67:35209)
0619 06:28:17.208933 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.208981 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.210888 (+ 1907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.211632 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.211651 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.212538 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:17.213574 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.213606 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.214966 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.214972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.217602 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.217619 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.218626 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.218636 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.218963 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.219673 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:17.219710 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:17.223479 (+ 3769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:17.227190 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:17.227201 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:17.227222 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:17.227954 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:17.228497 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:17.228506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:17.228515 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:17.228832 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:17.229726 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:17.229736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:17.230283 (+ 547us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.230676 (+ 393us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.231327 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":298,"threads_started":1}
I20260619 06:28:17.252595 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.238840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36519 (local address 127.0.97.126:32859)
0619 06:28:17.239004 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.239011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.239817 (+ 806us) server_negotiation.cc:407] Connection header received
0619 06:28:17.239990 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.239993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.240055 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.240130 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:17.241581 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.242468 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.243916 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.244349 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.244481 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.246601 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:17.246606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:17.246610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:17.246647 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:17.249736 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.250562 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.250566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.250568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.250673 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.251098 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.251105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.251110 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.251444 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:17.251592 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.252027 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.252255 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:28:17.252797 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.238929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:32859 (local address 127.0.97.67:36519)
0619 06:28:17.239700 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.239725 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.239880 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.240190 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.240197 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.240881 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:17.241401 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.241422 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.242611 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.242636 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.243782 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.243793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.244531 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.244538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.244701 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.245298 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:17.245317 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:17.246444 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:17.249873 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:17.249878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:17.249882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:17.250452 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:17.250781 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:17.250789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:17.250794 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:17.250942 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:17.251566 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:17.251571 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:17.251747 (+ 176us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.251919 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.252314 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":412,"threads_started":1}
I20260619 06:28:17.263404 8348 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:34719
I20260619 06:28:17.263425 8482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:34719 every 8 connection(s)
I20260619 06:28:17.264554 8348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:17.266495 8483 heartbeater.cc:347] Connected to a master server at 127.0.97.126:32859
I20260619 06:28:17.266667 8483 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:17.267225 8483 heartbeater.cc:511] Master 127.0.97.126:32859 requested a full tablet report, sending...
I20260619 06:28:17.268317 7998 ts_manager.cc:194] Registered new tserver with Master: 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719)
I20260619 06:28:17.269130 7998 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:35209
I20260619 06:28:17.270241 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 8348
I20260619 06:28:17.270423 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:28:17.270900 389 external_mini_cluster.cc:1428] Setting key 1cba69f07f2f8cc6a10b75cb00e8a0fc
I20260619 06:28:17.273480 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:28:17.301489 8205 heartbeater.cc:503] Master 127.0.97.126:32859 was elected leader, sending a full tablet report...
Jun 19 06:28:17 dist-test-slave-4t1f krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850497, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 06:28:17 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:17 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:28:17.323902 8092 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:17.314013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.0.97.126:32859)
0619 06:28:17.314243 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.314249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.314482 (+ 233us) server_negotiation.cc:407] Connection header received
0619 06:28:17.314618 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.314621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.314683 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.314770 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:17.315980 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.316680 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.318011 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.318413 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.318598 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.323693 (+ 5095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35766: BlockingRecv error: recv got EOF from 127.0.0.1:35766 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 06:28:17 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850497, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:17.339952 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.327060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.0.97.126:32859)
0619 06:28:17.327218 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.327223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.327599 (+ 376us) server_negotiation.cc:407] Connection header received
0619 06:28:17.327770 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.327774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.327831 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.327969 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:17.328965 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.329827 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.330986 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.331359 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.331447 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.334880 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:17.334884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:17.334888 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:17.334918 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:17.337507 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.338125 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.338137 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.338140 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.338255 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.338613 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.338618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.338622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.338877 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:17.339175 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.339441 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.339655 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:28:17.343801 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:28:17.350350 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:17.372066 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.362155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39293 (local address 127.0.0.1:44212)
0619 06:28:17.362982 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.363021 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.363210 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.364201 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.364210 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.364250 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:17.364704 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.364729 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.366606 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.366614 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.369273 (+ 2659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.369289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.371083 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.371091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.371333 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.371380 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.371466 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.371574 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":153,"thread_start_us":347,"threads_started":1}
I20260619 06:28:17.372207 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.363301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.0.97.65:34261)
0619 06:28:17.363871 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.363879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.363957 (+ 78us) server_negotiation.cc:407] Connection header received
0619 06:28:17.364181 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.364189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.364435 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.364625 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:17.365541 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.367086 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.369715 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.370677 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.370843 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.371064 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.371689 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.371874 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":286,"threads_started":1}
I20260619 06:28:17.373008 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.362690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34261 (local address 127.0.0.1:56178)
0619 06:28:17.363872 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.363889 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.364012 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.364837 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.364843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.364871 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:17.365340 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.365352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.367224 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.367229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.369565 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.369576 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.370896 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.370903 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.371200 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.371257 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.371396 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.371574 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"mutex_wait_us":31,"thread_start_us":1103,"threads_started":1}
I20260619 06:28:17.373005 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.362397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44212 (local address 127.0.97.66:39293)
0619 06:28:17.363614 (+ 1217us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.363625 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.363655 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:28:17.363766 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.363774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.364017 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.364215 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:17.364939 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.366462 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.369472 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.370823 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.371154 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.371360 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.371535 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.372694 (+ 1159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1042,"thread_start_us":925,"threads_started":1}
I20260619 06:28:17.373692 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.364735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:34719 (local address 127.0.0.1:45030)
0619 06:28:17.365406 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.365424 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.365520 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.366637 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.366642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.366669 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:17.367099 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.367113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.368929 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.368934 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.371116 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.371128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.372895 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.372901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.373124 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.373160 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.373228 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.373345 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":49,"thread_start_us":943,"threads_started":1}
I20260619 06:28:17.374584 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.365457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.0.97.67:34719)
0619 06:28:17.366108 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.366116 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.366144 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:28:17.366223 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.366230 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.366416 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.367297 (+ 881us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:17.367545 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.368782 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.371284 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.372677 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.374027 (+ 1350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.374206 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.374260 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.374357 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":662,"threads_started":1}
I20260619 06:28:17.376729 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 5035da8fa343437b80a86259864e7ba2 (DEFAULT_TABLE table=test-table [id=517be43c84de4612a5809482b360024f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:17.376729 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 5035da8fa343437b80a86259864e7ba2 (DEFAULT_TABLE table=test-table [id=517be43c84de4612a5809482b360024f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:17.377633 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 5035da8fa343437b80a86259864e7ba2 (DEFAULT_TABLE table=test-table [id=517be43c84de4612a5809482b360024f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:17.378201 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5035da8fa343437b80a86259864e7ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:17.378198 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5035da8fa343437b80a86259864e7ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:17.378738 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5035da8fa343437b80a86259864e7ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:17.389516 8507 tablet_bootstrap.cc:492] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Bootstrap starting.
I20260619 06:28:17.392872 8508 tablet_bootstrap.cc:492] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Bootstrap starting.
I20260619 06:28:17.393570 8509 tablet_bootstrap.cc:492] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Bootstrap starting.
I20260619 06:28:17.394272 8507 tablet_bootstrap.cc:654] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:17.395529 8507 log.cc:824] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:17.396916 8508 tablet_bootstrap.cc:654] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:17.398134 8507 tablet_bootstrap.cc:492] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: No bootstrap required, opened a new log
I20260619 06:28:17.398187 8508 log.cc:824] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:17.398423 8507 ts_tablet_manager.cc:1397] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260619 06:28:17.399241 8509 tablet_bootstrap.cc:654] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:17.400374 8509 log.cc:824] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:17.401141 8508 tablet_bootstrap.cc:492] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: No bootstrap required, opened a new log
I20260619 06:28:17.401379 8508 ts_tablet_manager.cc:1397] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260619 06:28:17.403439 8509 tablet_bootstrap.cc:492] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: No bootstrap required, opened a new log
I20260619 06:28:17.403681 8509 ts_tablet_manager.cc:1397] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260619 06:28:17.406665 8507 raft_consensus.cc:348] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.407042 8507 raft_consensus.cc:374] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:17.407176 8507 raft_consensus.cc:729] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Initialized, Role: FOLLOWER
I20260619 06:28:17.407851 8507 consensus_queue.cc:260] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.408975 8483 heartbeater.cc:503] Master 127.0.97.126:32859 was elected leader, sending a full tablet report...
I20260619 06:28:17.409543 8507 ts_tablet_manager.cc:1428] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260619 06:28:17.413782 8509 raft_consensus.cc:348] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.413847 8508 raft_consensus.cc:348] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.414219 8509 raft_consensus.cc:374] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:17.414219 8508 raft_consensus.cc:374] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:17.414324 8509 raft_consensus.cc:729] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92372700c0f4e13901a604976d34df5, State: Initialized, Role: FOLLOWER
I20260619 06:28:17.414320 8508 raft_consensus.cc:729] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326c5bf6dee74c8b89088a5b749c7bba, State: Initialized, Role: FOLLOWER
I20260619 06:28:17.414898 8509 consensus_queue.cc:260] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.414898 8508 consensus_queue.cc:260] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.415974 8344 heartbeater.cc:503] Master 127.0.97.126:32859 was elected leader, sending a full tablet report...
I20260619 06:28:17.416556 8508 ts_tablet_manager.cc:1428] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260619 06:28:17.416594 8509 ts_tablet_manager.cc:1428] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260619 06:28:17.516335 8484 tablet.cc:2369] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:28:17.523330 8345 tablet.cc:2369] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:28:17.544029 8206 tablet.cc:2369] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:28:17.735421 8515 raft_consensus.cc:482] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:17.735714 8515 raft_consensus.cc:504] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.737435 8513 raft_consensus.cc:482] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:17.737424 8515 leader_election.cc:288] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293), 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719)
I20260619 06:28:17.737658 8513 raft_consensus.cc:504] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.739372 8513 leader_election.cc:288] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293), d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261)
I20260619 06:28:17.747010 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.737965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36991 (local address 127.0.97.67:34719)
0619 06:28:17.738369 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.738377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.738625 (+ 248us) server_negotiation.cc:407] Connection header received
0619 06:28:17.738914 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.738920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.739013 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.739082 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:17.739818 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.741196 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.744537 (+ 3341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.745881 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.746025 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.746140 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.746640 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.746794 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269}
I20260619 06:28:17.747635 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.737680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:34719 (local address 127.0.97.65:36991)
0619 06:28:17.738503 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.738524 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.738738 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.739292 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.739296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.739324 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:17.739670 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.739685 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.741462 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.741470 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.744355 (+ 2885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.744367 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.746286 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.746293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.746446 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.746477 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.747243 (+ 766us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.747357 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":72,"thread_start_us":708,"threads_started":1}
I20260619 06:28:17.748534 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" is_pre_election: true
I20260619 06:28:17.748962 8502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.737883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49549 (local address 127.0.97.66:39293)
0619 06:28:17.738196 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.738206 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.739226 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 06:28:17.739435 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.739440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.739532 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.739604 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:17.743391 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.744442 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.746820 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.747899 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.748035 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.748129 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.748614 (+ 485us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.748733 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260619 06:28:17.748956 8437 raft_consensus.cc:2454] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92372700c0f4e13901a604976d34df5 in term 0.
I20260619 06:28:17.749087 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.737949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39293 (local address 127.0.97.65:49549)
0619 06:28:17.739098 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.739112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.739273 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.739805 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.739811 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.739833 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:17.743201 (+ 3368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.743213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.744622 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.744632 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.746684 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.746708 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.748270 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.748276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.748443 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.748472 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.748571 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.748697 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031,"mutex_wait_us":1,"thread_start_us":854,"threads_started":1}
I20260619 06:28:17.749766 8086 leader_election.cc:302] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b01b66c1e5f4621bb32fbba97fcf0a8, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:17.750206 8515 raft_consensus.cc:2781] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:17.749992 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba" is_pre_election: true
I20260619 06:28:17.750344 8515 raft_consensus.cc:482] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:17.750447 8515 raft_consensus.cc:3037] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:17.750609 8294 raft_consensus.cc:2454] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92372700c0f4e13901a604976d34df5 in term 0.
I20260619 06:28:17.751791 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.740120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54391 (local address 127.0.97.66:39293)
0619 06:28:17.742051 (+ 1931us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.742059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.742577 (+ 518us) server_negotiation.cc:407] Connection header received
0619 06:28:17.742867 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.742874 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.743008 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.743073 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:17.744241 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.745540 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.749188 (+ 3648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.750597 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.750698 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.750825 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.751341 (+ 516us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.751481 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1766,"thread_start_us":1498,"threads_started":1}
I20260619 06:28:17.752075 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.740501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35009 (local address 127.0.97.65:34261)
0619 06:28:17.741836 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.741844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.742925 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 06:28:17.743142 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.743152 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.743242 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.743301 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:17.744223 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.745521 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.749685 (+ 4164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.750973 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.751078 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.751177 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.751709 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.751844 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":62}
I20260619 06:28:17.752146 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.739786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34261 (local address 127.0.97.67:35009)
0619 06:28:17.742848 (+ 3062us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.742862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.742998 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.743476 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.743481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.743513 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:17.744051 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.744062 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.745703 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.745707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.748086 (+ 2379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.748099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.751361 (+ 3262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.751366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.751557 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.751584 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.751671 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.751787 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2955,"mutex_wait_us":137,"thread_start_us":2450,"threads_started":1}
I20260619 06:28:17.752758 8519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.739614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39293 (local address 127.0.97.67:54391)
0619 06:28:17.742467 (+ 2853us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.742484 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.742679 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.743340 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.743344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.743366 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:17.743972 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.743988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.745704 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.745708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.748979 (+ 3271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.748994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.750986 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.750995 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.751163 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.751195 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.751926 (+ 731us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.752047 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2714,"mutex_wait_us":1273,"spinlock_wait_cycles":19200,"thread_start_us":2970,"threads_started":1}
I20260619 06:28:17.752933 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5" is_pre_election: true
I20260619 06:28:17.753389 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba" is_pre_election: true
I20260619 06:28:17.753634 8294 raft_consensus.cc:2454] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 0.
I20260619 06:28:17.753607 8515 raft_consensus.cc:504] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.754020 8160 raft_consensus.cc:2379] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in current term 1: Already voted for candidate d92372700c0f4e13901a604976d34df5 in this term.
I20260619 06:28:17.754055 8366 leader_election.cc:302] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326c5bf6dee74c8b89088a5b749c7bba, 5b01b66c1e5f4621bb32fbba97fcf0a8; no voters:
I20260619 06:28:17.754339 8515 leader_election.cc:288] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 election: Requested vote from peers 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293), 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719)
I20260619 06:28:17.754595 8513 raft_consensus.cc:2781] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:17.754751 8513 raft_consensus.cc:482] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:17.754760 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba"
I20260619 06:28:17.754879 8513 raft_consensus.cc:3037] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:17.754889 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8"
I20260619 06:28:17.754935 8294 raft_consensus.cc:3037] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:17.757052 8294 raft_consensus.cc:2454] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92372700c0f4e13901a604976d34df5 in term 1.
I20260619 06:28:17.757334 8513 raft_consensus.cc:504] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.757475 8087 leader_election.cc:302] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326c5bf6dee74c8b89088a5b749c7bba, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:17.757673 8437 raft_consensus.cc:2379] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d92372700c0f4e13901a604976d34df5 in current term 1: Already voted for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in this term.
I20260619 06:28:17.757709 8515 raft_consensus.cc:2781] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:17.757911 8513 leader_election.cc:288] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Requested vote from peers 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293), d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261)
I20260619 06:28:17.758319 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba"
I20260619 06:28:17.758687 8294 raft_consensus.cc:2379] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in current term 1: Already voted for candidate d92372700c0f4e13901a604976d34df5 in this term.
I20260619 06:28:17.758677 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5035da8fa343437b80a86259864e7ba2" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5"
I20260619 06:28:17.758935 8515 raft_consensus.cc:686] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 1 LEADER]: Becoming Leader. State: Replica: d92372700c0f4e13901a604976d34df5, State: Running, Role: LEADER
I20260619 06:28:17.759639 8515 consensus_queue.cc:237] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.763036 8367 leader_election.cc:302] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b01b66c1e5f4621bb32fbba97fcf0a8; no voters: 326c5bf6dee74c8b89088a5b749c7bba, d92372700c0f4e13901a604976d34df5
I20260619 06:28:17.763330 8513 raft_consensus.cc:2726] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 06:28:17.763929 7998 catalog_manager.cc:5495] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d92372700c0f4e13901a604976d34df5 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "d92372700c0f4e13901a604976d34df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20260619 06:28:17.825516 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.820531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.0.97.65:34261)
0619 06:28:17.820711 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.820716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.820734 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:28:17.820886 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.820892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.820991 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.821173 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:17.821520 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.822242 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.823502 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.823895 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.824035 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.824433 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:17.824920 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:17.825124 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.825173 (+ 49us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.825247 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:28:17.832760 8437 raft_consensus.cc:1261] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Refusing update from remote peer d92372700c0f4e13901a604976d34df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:17.832865 8294 raft_consensus.cc:1261] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Refusing update from remote peer d92372700c0f4e13901a604976d34df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:17.833557 8521 consensus_queue.cc:1035] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:17.833912 8515 consensus_queue.cc:1035] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:17.844496 8526 mvcc.cc:205] Tried to move back new op lower bound from 7298459639114936320 to 7298459638826921984. Current Snapshot: MvccSnapshot[applied={T|T < 7298459639114936320}]
I20260619 06:28:17.845345 8524 mvcc.cc:205] Tried to move back new op lower bound from 7298459639114936320 to 7298459638826921984. Current Snapshot: MvccSnapshot[applied={T|T < 7298459639114936320}]
I20260619 06:28:17.863358 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:17.863624 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:17.866948 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Sending DeleteTablet for 3 replicas of tablet 5035da8fa343437b80a86259864e7ba2
I20260619 06:28:17.867847 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 5035da8fa343437b80a86259864e7ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:17 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44212
I20260619 06:28:17.867973 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet 5035da8fa343437b80a86259864e7ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:17 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:56178
I20260619 06:28:17.868239 8413 tablet_service.cc:1514] Processing DeleteTablet for tablet 5035da8fa343437b80a86259864e7ba2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:17 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45030
I20260619 06:28:17.870251 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35782:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:28:17.871048 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:17.880380 8534 tablet_replica.cc:330] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: stopping tablet replica
I20260619 06:28:17.880867 8534 raft_consensus.cc:2229] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:28:17.881600 8534 raft_consensus.cc:2258] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:17.882157 8533 tablet_replica.cc:330] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: stopping tablet replica
I20260619 06:28:17.882635 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee0b6637cd94da5bbeb7581039a643c (DEFAULT_TABLE table=test-table [id=bf36fe74395a424c84dd81b0a8e5ea9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:17.883375 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee0b6637cd94da5bbeb7581039a643c (DEFAULT_TABLE table=test-table [id=bf36fe74395a424c84dd81b0a8e5ea9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:17.883848 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee0b6637cd94da5bbeb7581039a643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:17.884370 8534 ts_tablet_manager.cc:1905] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:17.885423 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 5ee0b6637cd94da5bbeb7581039a643c (DEFAULT_TABLE table=test-table [id=bf36fe74395a424c84dd81b0a8e5ea9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:17.886197 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee0b6637cd94da5bbeb7581039a643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:17.890108 8507 tablet_bootstrap.cc:492] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: Bootstrap starting.
I20260619 06:28:17.891706 8534 ts_tablet_manager.cc:1918] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:17.891855 8534 log.cc:1196] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/5035da8fa343437b80a86259864e7ba2
I20260619 06:28:17.892218 8507 tablet_bootstrap.cc:654] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:17.892385 8534 ts_tablet_manager.cc:1939] T 5035da8fa343437b80a86259864e7ba2 P d92372700c0f4e13901a604976d34df5: Deleting consensus metadata
I20260619 06:28:17.882795 8533 raft_consensus.cc:2229] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:17.893568 8533 raft_consensus.cc:2258] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:17.883111 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ee0b6637cd94da5bbeb7581039a643c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:17.896960 8509 tablet_bootstrap.cc:492] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: Bootstrap starting.
I20260619 06:28:17.898542 8509 tablet_bootstrap.cc:654] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:17.901897 8535 tablet_replica.cc:330] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: stopping tablet replica
I20260619 06:28:17.902304 8535 raft_consensus.cc:2229] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:17.902550 8507 tablet_bootstrap.cc:492] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: No bootstrap required, opened a new log
I20260619 06:28:17.902668 8507 ts_tablet_manager.cc:1397] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20260619 06:28:17.902781 8535 raft_consensus.cc:2258] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:17.903466 8507 raft_consensus.cc:348] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.903687 8507 raft_consensus.cc:374] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:17.903782 8507 raft_consensus.cc:729] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Initialized, Role: FOLLOWER
I20260619 06:28:17.904037 8507 consensus_queue.cc:260] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.904476 8533 ts_tablet_manager.cc:1905] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:17.904556 8507 ts_tablet_manager.cc:1428] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260619 06:28:17.906210 8535 ts_tablet_manager.cc:1905] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:17.911629 8508 tablet_bootstrap.cc:492] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: Bootstrap starting.
I20260619 06:28:17.913277 8508 tablet_bootstrap.cc:654] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:17.913733 8535 ts_tablet_manager.cc:1918] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:17.913885 8535 log.cc:1196] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/5035da8fa343437b80a86259864e7ba2
I20260619 06:28:17.914402 8535 ts_tablet_manager.cc:1939] T 5035da8fa343437b80a86259864e7ba2 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting consensus metadata
I20260619 06:28:17.915447 8509 tablet_bootstrap.cc:492] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: No bootstrap required, opened a new log
I20260619 06:28:17.915637 8509 ts_tablet_manager.cc:1397] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.005s
I20260619 06:28:17.916029 7983 catalog_manager.cc:4841] TS 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719): tablet 5035da8fa343437b80a86259864e7ba2 (table test-table [id=517be43c84de4612a5809482b360024f]) successfully deleted
I20260619 06:28:17.916625 8509 raft_consensus.cc:348] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.916952 8509 raft_consensus.cc:374] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:17.917122 8509 raft_consensus.cc:729] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92372700c0f4e13901a604976d34df5, State: Initialized, Role: FOLLOWER
I20260619 06:28:17.917477 8509 consensus_queue.cc:260] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.918210 8509 ts_tablet_manager.cc:1428] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:28:17.918282 8513 raft_consensus.cc:482] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:17.918447 8513 raft_consensus.cc:504] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.919054 8513 leader_election.cc:288] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261), 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293)
I20260619 06:28:17.919946 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee0b6637cd94da5bbeb7581039a643c" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5" is_pre_election: true
I20260619 06:28:17.920044 8533 ts_tablet_manager.cc:1918] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:17.920184 8533 log.cc:1196] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/5035da8fa343437b80a86259864e7ba2
I20260619 06:28:17.920239 8160 raft_consensus.cc:2454] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 0.
I20260619 06:28:17.920658 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee0b6637cd94da5bbeb7581039a643c" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba" is_pre_election: true
I20260619 06:28:17.921039 8367 leader_election.cc:302] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b01b66c1e5f4621bb32fbba97fcf0a8, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:17.921145 7985 catalog_manager.cc:4841] TS d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261): tablet 5035da8fa343437b80a86259864e7ba2 (table test-table [id=517be43c84de4612a5809482b360024f]) successfully deleted
W20260619 06:28:17.921490 8366 leader_election.cc:341] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293): Illegal state: must be running to vote when last-logged opid is not known
I20260619 06:28:17.921864 8513 raft_consensus.cc:2781] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:17.921969 8513 raft_consensus.cc:482] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:17.920765 8533 ts_tablet_manager.cc:1939] T 5035da8fa343437b80a86259864e7ba2 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting consensus metadata
I20260619 06:28:17.922091 8513 raft_consensus.cc:3037] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:17.923405 8508 tablet_bootstrap.cc:492] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: No bootstrap required, opened a new log
I20260619 06:28:17.923799 7984 catalog_manager.cc:4841] TS 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293): tablet 5035da8fa343437b80a86259864e7ba2 (table test-table [id=517be43c84de4612a5809482b360024f]) successfully deleted
I20260619 06:28:17.923555 8508 ts_tablet_manager.cc:1397] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260619 06:28:17.924708 8508 raft_consensus.cc:348] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.924782 8513 raft_consensus.cc:504] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.924921 8508 raft_consensus.cc:374] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:17.925053 8508 raft_consensus.cc:729] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326c5bf6dee74c8b89088a5b749c7bba, State: Initialized, Role: FOLLOWER
I20260619 06:28:17.925390 8508 consensus_queue.cc:260] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.925774 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee0b6637cd94da5bbeb7581039a643c" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5"
I20260619 06:28:17.925983 8160 raft_consensus.cc:3037] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:17.926067 8508 ts_tablet_manager.cc:1428] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260619 06:28:17.928985 8160 raft_consensus.cc:2454] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 1.
I20260619 06:28:17.929338 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ee0b6637cd94da5bbeb7581039a643c" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba"
I20260619 06:28:17.929495 8367 leader_election.cc:302] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b01b66c1e5f4621bb32fbba97fcf0a8, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:17.929544 8294 raft_consensus.cc:3037] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:17.930116 8539 raft_consensus.cc:2781] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:17.933086 8294 raft_consensus.cc:2454] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 1.
I20260619 06:28:17.933182 8513 leader_election.cc:288] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Requested vote from peers d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261), 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293)
I20260619 06:28:17.933815 8539 raft_consensus.cc:686] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 LEADER]: Becoming Leader. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Running, Role: LEADER
I20260619 06:28:17.934195 8539 consensus_queue.cc:237] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:17.937796 7998 catalog_manager.cc:5495] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } health_report { overall_health: HEALTHY } } }
Jun 19 06:28:17 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850495, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.126@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:17.985253 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.967114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:32859 (local address 127.0.0.1:35784)
0619 06:28:17.967777 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:17.967801 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:17.967948 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:17.968271 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:17.968275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:17.968896 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:17.969158 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.969168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.970151 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.970159 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.971075 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:17.971086 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.971732 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.971738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:17.971881 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.977806 (+ 5925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:17.977825 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:17.979853 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:17.982812 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:17.982823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:17.982830 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:17.983198 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:17.983552 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:17.983556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:17.983560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:17.983691 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:17.984179 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:17.984186 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:17.984351 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 06:28:17.984505 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 06:28:17.984784 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":353,"threads_started":1}
I20260619 06:28:17.985309 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:17.967136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.0.97.126:32859)
0619 06:28:17.968040 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:17.968046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:17.968068 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:28:17.968105 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:17.968108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:17.968152 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:17.968256 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:17.969296 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.970030 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.971207 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:17.971582 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:17.971713 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:17.980059 (+ 8346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:17.980064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:17.980067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:17.980093 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:17.982673 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.983311 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.983315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.983319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.983442 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:17.983791 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:17.983802 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:17.983805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:17.984057 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:17.984203 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:17.984589 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 06:28:17.984902 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":783,"thread_start_us":638,"threads_started":1}
I20260619 06:28:17.990052 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35784:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 06:28:17.991166 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:17.997690 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 04e00a38397541d0a1eb57c505df995e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ded5bf7c0e44110bbc2a3ef15e6d0d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:28:17.997927 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 04e00a38397541d0a1eb57c505df995e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ded5bf7c0e44110bbc2a3ef15e6d0d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:28:17.998296 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e00a38397541d0a1eb57c505df995e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:17.998443 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e00a38397541d0a1eb57c505df995e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.003763 8508 tablet_bootstrap.cc:492] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: Bootstrap starting.
I20260619 06:28:18.004385 8413 tablet_service.cc:1467] Processing CreateTablet for tablet 04e00a38397541d0a1eb57c505df995e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6ded5bf7c0e44110bbc2a3ef15e6d0d1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 06:28:18.004886 8413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e00a38397541d0a1eb57c505df995e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.005170 8508 tablet_bootstrap.cc:654] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.008251 8509 tablet_bootstrap.cc:492] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: Bootstrap starting.
I20260619 06:28:18.009809 8509 tablet_bootstrap.cc:654] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.010232 8508 tablet_bootstrap.cc:492] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: No bootstrap required, opened a new log
I20260619 06:28:18.010704 8508 ts_tablet_manager.cc:1397] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260619 06:28:18.011754 8508 raft_consensus.cc:348] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.012216 8508 raft_consensus.cc:374] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.012393 8508 raft_consensus.cc:729] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326c5bf6dee74c8b89088a5b749c7bba, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.012697 8507 tablet_bootstrap.cc:492] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: Bootstrap starting.
I20260619 06:28:18.012857 8508 consensus_queue.cc:260] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.013655 8508 ts_tablet_manager.cc:1428] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:28:18.013487 8514 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.013840 8514 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:28:18.014285 8507 tablet_bootstrap.cc:654] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.022753 8509 tablet_bootstrap.cc:492] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: No bootstrap required, opened a new log
I20260619 06:28:18.022912 8509 ts_tablet_manager.cc:1397] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.006s
I20260619 06:28:18.023077 8507 tablet_bootstrap.cc:492] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: No bootstrap required, opened a new log
I20260619 06:28:18.023244 8507 ts_tablet_manager.cc:1397] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260619 06:28:18.023653 8509 raft_consensus.cc:348] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.023895 8509 raft_consensus.cc:374] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.023808 8507 raft_consensus.cc:348] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.023985 8509 raft_consensus.cc:729] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92372700c0f4e13901a604976d34df5, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.024080 8507 raft_consensus.cc:374] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.024214 8507 raft_consensus.cc:729] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.024212 8509 consensus_queue.cc:260] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.024432 8507 consensus_queue.cc:260] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.024937 8507 ts_tablet_manager.cc:1428] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 06:28:18.024921 8509 ts_tablet_manager.cc:1428] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260619 06:28:18.024820 8515 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.025501 8539 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.025724 8515 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:28:18.025774 8539 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:28:18.090116 8539 raft_consensus.cc:482] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:18.090342 8539 raft_consensus.cc:504] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.091045 8539 leader_election.cc:288] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261), 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293)
I20260619 06:28:18.091657 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e00a38397541d0a1eb57c505df995e" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba" is_pre_election: true
I20260619 06:28:18.091693 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e00a38397541d0a1eb57c505df995e" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5" is_pre_election: true
I20260619 06:28:18.091931 8294 raft_consensus.cc:2454] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 0.
I20260619 06:28:18.091945 8160 raft_consensus.cc:2454] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 0.
I20260619 06:28:18.092370 8366 leader_election.cc:302] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326c5bf6dee74c8b89088a5b749c7bba, 5b01b66c1e5f4621bb32fbba97fcf0a8; no voters:
I20260619 06:28:18.092682 8539 raft_consensus.cc:2781] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:18.092789 8539 raft_consensus.cc:482] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:18.092895 8539 raft_consensus.cc:3037] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:18.094902 8539 raft_consensus.cc:504] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.095872 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e00a38397541d0a1eb57c505df995e" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5"
I20260619 06:28:18.096101 8160 raft_consensus.cc:3037] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:18.096067 8294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e00a38397541d0a1eb57c505df995e" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba"
I20260619 06:28:18.096278 8294 raft_consensus.cc:3037] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:18.096304 8539 leader_election.cc:288] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Requested vote from peers d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261), 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293)
I20260619 06:28:18.099144 8294 raft_consensus.cc:2454] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 1.
I20260619 06:28:18.099161 8160 raft_consensus.cc:2454] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 1.
I20260619 06:28:18.099539 8366 leader_election.cc:302] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326c5bf6dee74c8b89088a5b749c7bba, 5b01b66c1e5f4621bb32fbba97fcf0a8; no voters:
I20260619 06:28:18.099860 8539 raft_consensus.cc:2781] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:18.100054 8539 raft_consensus.cc:686] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 LEADER]: Becoming Leader. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Running, Role: LEADER
I20260619 06:28:18.100297 8539 consensus_queue.cc:237] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.101982 8540 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: TxnStatusTablet state changed. Reason: New leader 5b01b66c1e5f4621bb32fbba97fcf0a8. Latest consensus state: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.102360 8540 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:28:18.103050 8553 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 06:28:18.103288 8553 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 06:28:18.103579 7998 catalog_manager.cc:5495] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } health_report { overall_health: HEALTHY } } }
I20260619 06:28:18.135552 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.128671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.0.97.67:34719)
0619 06:28:18.128858 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.128865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.128888 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:28:18.129023 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.129028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.129164 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.129302 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:18.129737 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.130956 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.132898 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.133542 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.133676 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.134202 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:18.134897 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:18.135047 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:18.135151 (+ 104us) server_negotiation.cc:299] Negotiation successful
0619 06:28:18.135230 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:28:18.135701 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.128329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:34719 (local address 127.0.0.1:45044)
0619 06:28:18.128616 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:18.128639 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:18.128775 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:18.129360 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:18.129365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:18.129379 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:28:18.129581 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.129593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.131119 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.131125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.132741 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.132755 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.133704 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.133711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.133921 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.133992 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:28:18.135013 (+ 1021us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:28:18.135032 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 06:28:18.135131 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 06:28:18.135334 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:28:18.139889 8294 raft_consensus.cc:1261] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Refusing update from remote peer 5b01b66c1e5f4621bb32fbba97fcf0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:18.140666 8160 raft_consensus.cc:1261] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Refusing update from remote peer 5b01b66c1e5f4621bb32fbba97fcf0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:18.140564 8540 consensus_queue.cc:1035] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:18.141386 8540 consensus_queue.cc:1035] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:18.149627 8514 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: TxnStatusTablet state changed. Reason: New leader 5b01b66c1e5f4621bb32fbba97fcf0a8. Latest consensus state: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.150079 8514 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:28:18.152371 8540 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.152606 8540 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:28:18.152781 8525 mvcc.cc:205] Tried to move back new op lower bound from 7298459640374538240 to 7298459640220688384. Current Snapshot: MvccSnapshot[applied={T|T < 7298459640374538240 or (T in {7298459640374538240})}]
I20260619 06:28:18.154418 8515 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: TxnStatusTablet state changed. Reason: New leader 5b01b66c1e5f4621bb32fbba97fcf0a8. Latest consensus state: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.154690 8515 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:28:18.158059 8514 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.158293 8514 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P 326c5bf6dee74c8b89088a5b749c7bba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:28:18.160331 8513 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.160660 8513 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P 5b01b66c1e5f4621bb32fbba97fcf0a8: This TxnStatusTablet replica's current role is: LEADER
I20260619 06:28:18.162801 8515 tablet_replica.cc:439] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } } }
I20260619 06:28:18.163038 8515 tablet_replica.cc:442] T 04e00a38397541d0a1eb57c505df995e P d92372700c0f4e13901a604976d34df5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 06:28:18.170717 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.164093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45050 (local address 127.0.97.67:34719)
0619 06:28:18.164257 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.164263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.164546 (+ 283us) server_negotiation.cc:407] Connection header received
0619 06:28:18.164760 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.164766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.164846 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.165454 (+ 608us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:18.165602 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.166704 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.168589 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.169295 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.169442 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.169959 (+ 517us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:18.170090 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:18.170192 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:18.170355 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 06:28:18.170429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 06:28:18.189150 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.182126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36023 (local address 127.0.97.67:34719)
0619 06:28:18.182298 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.182305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.182565 (+ 260us) server_negotiation.cc:407] Connection header received
0619 06:28:18.182783 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.182788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.182909 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.183015 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:18.183641 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.184759 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.186970 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.187697 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.187825 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.188308 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:18.188463 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:18.188563 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:18.188709 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:28:18.188789 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:28:18.189260 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.181746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:34719 (local address 127.0.97.67:36023)
0619 06:28:18.182472 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:18.182493 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:18.182640 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:18.183044 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:18.183049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:18.183062 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:28:18.183460 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.183471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.184905 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.184911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.186789 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.186804 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.187887 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.187894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.188059 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.188133 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:28:18.188577 (+ 444us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:28:18.188587 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:28:18.188670 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 06:28:18.188820 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":556,"threads_started":1}
I20260619 06:28:18.199709 8293 raft_consensus.cc:1261] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Refusing update from remote peer 5b01b66c1e5f4621bb32fbba97fcf0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:18.199728 8160 raft_consensus.cc:1261] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Refusing update from remote peer 5b01b66c1e5f4621bb32fbba97fcf0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:18.200305 8540 consensus_queue.cc:1035] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:18.200601 8513 consensus_queue.cc:1035] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:18.322301 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:18.322556 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35782:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:18.326267 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Sending DeleteTablet for 3 replicas of tablet 5ee0b6637cd94da5bbeb7581039a643c
I20260619 06:28:18.326972 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee0b6637cd94da5bbeb7581039a643c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:18 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:56178
I20260619 06:28:18.327209 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee0b6637cd94da5bbeb7581039a643c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:18 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44212
I20260619 06:28:18.327498 8533 tablet_replica.cc:330] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: stopping tablet replica
I20260619 06:28:18.327721 8533 raft_consensus.cc:2229] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:18.327746 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ee0b6637cd94da5bbeb7581039a643c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:18 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45030
I20260619 06:28:18.327915 8533 raft_consensus.cc:2258] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:18.327997 8535 tablet_replica.cc:330] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: stopping tablet replica
I20260619 06:28:18.328194 8535 raft_consensus.cc:2229] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:28:18.328651 8535 raft_consensus.cc:2258] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:18.329167 8533 ts_tablet_manager.cc:1905] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:18.330271 8534 tablet_replica.cc:330] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: stopping tablet replica
I20260619 06:28:18.330436 8534 raft_consensus.cc:2229] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:18.330610 8534 raft_consensus.cc:2258] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:18.332118 8534 ts_tablet_manager.cc:1905] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:18.337779 8535 ts_tablet_manager.cc:1905] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:18.341979 8533 ts_tablet_manager.cc:1918] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:28:18.344789 8534 ts_tablet_manager.cc:1918] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:28:18.344985 8533 log.cc:1196] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/5ee0b6637cd94da5bbeb7581039a643c
I20260619 06:28:18.345759 8533 ts_tablet_manager.cc:1939] T 5ee0b6637cd94da5bbeb7581039a643c P 326c5bf6dee74c8b89088a5b749c7bba: Deleting consensus metadata
I20260619 06:28:18.345945 8534 log.cc:1196] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/5ee0b6637cd94da5bbeb7581039a643c
I20260619 06:28:18.346437 8534 ts_tablet_manager.cc:1939] T 5ee0b6637cd94da5bbeb7581039a643c P d92372700c0f4e13901a604976d34df5: Deleting consensus metadata
I20260619 06:28:18.347558 7984 catalog_manager.cc:4841] TS 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293): tablet 5ee0b6637cd94da5bbeb7581039a643c (table test-table [id=bf36fe74395a424c84dd81b0a8e5ea9f]) successfully deleted
I20260619 06:28:18.348856 7985 catalog_manager.cc:4841] TS d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261): tablet 5ee0b6637cd94da5bbeb7581039a643c (table test-table [id=bf36fe74395a424c84dd81b0a8e5ea9f]) successfully deleted
I20260619 06:28:18.348937 8535 ts_tablet_manager.cc:1918] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:28:18.349102 8535 log.cc:1196] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/5ee0b6637cd94da5bbeb7581039a643c
I20260619 06:28:18.349537 8535 ts_tablet_manager.cc:1939] T 5ee0b6637cd94da5bbeb7581039a643c P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting consensus metadata
I20260619 06:28:18.350490 7983 catalog_manager.cc:4841] TS 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719): tablet 5ee0b6637cd94da5bbeb7581039a643c (table test-table [id=bf36fe74395a424c84dd81b0a8e5ea9f]) successfully deleted
Jun 19 06:28:18 dist-test-slave-4t1f krb5kdc[7953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 06:28:18 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:18 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:28:18.407269 8548 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:18.395022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35788 (local address 127.0.97.126:32859)
0619 06:28:18.395247 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.395254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.395779 (+ 525us) server_negotiation.cc:407] Connection header received
0619 06:28:18.395952 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.395957 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.396030 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.396131 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:18.397729 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.398861 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.400627 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.401247 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.401355 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.406636 (+ 5281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35788: BlockingRecv error: recv got EOF from 127.0.0.1:35788 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 06:28:18 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM
I20260619 06:28:18.424326 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.409291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35794 (local address 127.0.97.126:32859)
0619 06:28:18.409452 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.409457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.409777 (+ 320us) server_negotiation.cc:407] Connection header received
0619 06:28:18.409888 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.409893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.409959 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.410048 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:18.411263 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.412358 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.413555 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.414232 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.414356 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.418482 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:18.418489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:18.418493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:18.418531 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:18.421934 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:18.422487 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:18.422493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:18.422497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:18.422662 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:18.422995 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:18.423002 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:18.423005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:18.423326 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:18.423580 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:18.423750 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 06:28:18.423959 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:28:18.427556 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:28:18.428177 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:18.434254 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 3263dd64f7704710bee8c09cd7947a44 (DEFAULT_TABLE table=test-table [id=65bbd39e53a54dafb1e10863c889f760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:18.434512 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 3263dd64f7704710bee8c09cd7947a44 (DEFAULT_TABLE table=test-table [id=65bbd39e53a54dafb1e10863c889f760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:18.434710 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3263dd64f7704710bee8c09cd7947a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.434921 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3263dd64f7704710bee8c09cd7947a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.434846 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 3263dd64f7704710bee8c09cd7947a44 (DEFAULT_TABLE table=test-table [id=65bbd39e53a54dafb1e10863c889f760]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:18.435240 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3263dd64f7704710bee8c09cd7947a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.441288 8509 tablet_bootstrap.cc:492] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: Bootstrap starting.
I20260619 06:28:18.441368 8507 tablet_bootstrap.cc:492] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Bootstrap starting.
I20260619 06:28:18.441684 8508 tablet_bootstrap.cc:492] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Bootstrap starting.
I20260619 06:28:18.442834 8509 tablet_bootstrap.cc:654] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.442873 8507 tablet_bootstrap.cc:654] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.443182 8508 tablet_bootstrap.cc:654] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.450918 8508 tablet_bootstrap.cc:492] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: No bootstrap required, opened a new log
I20260619 06:28:18.451088 8508 ts_tablet_manager.cc:1397] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:28:18.451810 8508 raft_consensus.cc:348] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.452023 8507 tablet_bootstrap.cc:492] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: No bootstrap required, opened a new log
I20260619 06:28:18.452064 8508 raft_consensus.cc:374] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.452164 8508 raft_consensus.cc:729] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326c5bf6dee74c8b89088a5b749c7bba, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.452157 8507 ts_tablet_manager.cc:1397] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20260619 06:28:18.452405 8508 consensus_queue.cc:260] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.453111 8508 ts_tablet_manager.cc:1428] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 06:28:18.453155 8507 raft_consensus.cc:348] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.453471 8507 raft_consensus.cc:374] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.453578 8507 raft_consensus.cc:729] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.453812 8507 consensus_queue.cc:260] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.454409 8507 ts_tablet_manager.cc:1428] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260619 06:28:18.455092 8509 tablet_bootstrap.cc:492] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: No bootstrap required, opened a new log
I20260619 06:28:18.455232 8509 ts_tablet_manager.cc:1397] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20260619 06:28:18.456760 8509 raft_consensus.cc:348] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.457003 8509 raft_consensus.cc:374] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.457136 8509 raft_consensus.cc:729] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92372700c0f4e13901a604976d34df5, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.457396 8509 consensus_queue.cc:260] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.458469 8509 ts_tablet_manager.cc:1428] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260619 06:28:18.697853 8513 raft_consensus.cc:482] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:18.698053 8513 raft_consensus.cc:504] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.698761 8513 leader_election.cc:288] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261), 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293)
I20260619 06:28:18.698791 8514 raft_consensus.cc:482] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:18.698943 8514 raft_consensus.cc:504] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.699705 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3263dd64f7704710bee8c09cd7947a44" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5" is_pre_election: true
I20260619 06:28:18.699944 8160 raft_consensus.cc:2454] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 0.
I20260619 06:28:18.700457 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3263dd64f7704710bee8c09cd7947a44" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba" is_pre_election: true
I20260619 06:28:18.700582 8514 leader_election.cc:288] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261), 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719)
I20260619 06:28:18.700688 8293 raft_consensus.cc:2454] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 0.
I20260619 06:28:18.701014 8367 leader_election.cc:302] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b01b66c1e5f4621bb32fbba97fcf0a8, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:18.701364 8513 raft_consensus.cc:2781] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:18.701489 8513 raft_consensus.cc:482] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:18.701593 8513 raft_consensus.cc:3037] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:18.705458 8513 raft_consensus.cc:504] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.706629 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3263dd64f7704710bee8c09cd7947a44" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5"
I20260619 06:28:18.706898 8160 raft_consensus.cc:3037] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:18.710767 8160 raft_consensus.cc:2454] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 1.
I20260619 06:28:18.711360 8367 leader_election.cc:302] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b01b66c1e5f4621bb32fbba97fcf0a8, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:18.711674 8540 raft_consensus.cc:2781] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:18.711862 8540 raft_consensus.cc:686] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 LEADER]: Becoming Leader. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Running, Role: LEADER
I20260619 06:28:18.712024 8513 leader_election.cc:288] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [CANDIDATE]: Term 1 election: Requested vote from peers d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261), 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293)
I20260619 06:28:18.712122 8540 consensus_queue.cc:237] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.720430 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3263dd64f7704710bee8c09cd7947a44" candidate_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba"
I20260619 06:28:18.720675 8293 raft_consensus.cc:3037] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:18.722898 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.701833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34261 (local address 127.0.97.66:33601)
0619 06:28:18.713889 (+ 12056us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:18.713916 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:18.714028 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:18.714886 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:18.714890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:18.714920 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:18.715242 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.715257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.717288 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.717296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.720148 (+ 2852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.720161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.721959 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.721965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.722171 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.722209 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:28:18.722294 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:28:18.722430 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11930,"thread_start_us":11830,"threads_started":1}
I20260619 06:28:18.723625 8293 raft_consensus.cc:2454] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in term 1.
I20260619 06:28:18.723929 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.702143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33601 (local address 127.0.97.65:34261)
0619 06:28:18.714516 (+ 12373us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.714524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.714547 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:28:18.714620 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.714628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.714719 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.715436 (+ 717us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:18.715613 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.717109 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.720329 (+ 3220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.721647 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.721770 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.721861 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:18.723394 (+ 1533us) server_negotiation.cc:299] Negotiation successful
0619 06:28:18.723554 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12218,"thread_start_us":12584,"threads_started":1}
I20260619 06:28:18.724606 8160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3263dd64f7704710bee8c09cd7947a44" candidate_uuid: "326c5bf6dee74c8b89088a5b749c7bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92372700c0f4e13901a604976d34df5" is_pre_election: true
I20260619 06:28:18.724861 8160 raft_consensus.cc:2379] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 326c5bf6dee74c8b89088a5b749c7bba in current term 1: Already voted for candidate 5b01b66c1e5f4621bb32fbba97fcf0a8 in this term.
I20260619 06:28:18.727463 7998 catalog_manager.cc:5495] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } health_report { overall_health: HEALTHY } } }
I20260619 06:28:18.730511 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.701134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:34719 (local address 127.0.97.66:47949)
0619 06:28:18.713304 (+ 12170us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:18.713327 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:18.713456 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:18.724411 (+ 10955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:18.724415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:18.724440 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:18.724719 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.724731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.726310 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.726316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.728168 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:18.728181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.729787 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.729793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:18.729964 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.729985 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:28:18.730085 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:28:18.730215 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11867,"thread_start_us":11776,"threads_started":1}
I20260619 06:28:18.730537 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.701453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:47949 (local address 127.0.97.67:34719)
0619 06:28:18.719763 (+ 18310us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.719769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.719786 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:28:18.719837 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.719842 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.719900 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.724875 (+ 4975us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:18.725046 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.726144 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.728377 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.729630 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.729771 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.729855 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:18.730107 (+ 252us) server_negotiation.cc:299] Negotiation successful
0619 06:28:18.730229 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18185,"thread_start_us":18517,"threads_started":1}
I20260619 06:28:18.731168 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3263dd64f7704710bee8c09cd7947a44" candidate_uuid: "326c5bf6dee74c8b89088a5b749c7bba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" is_pre_election: true
I20260619 06:28:18.731699 8226 leader_election.cc:302] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 326c5bf6dee74c8b89088a5b749c7bba; no voters: 5b01b66c1e5f4621bb32fbba97fcf0a8, d92372700c0f4e13901a604976d34df5
I20260619 06:28:18.732028 8514 raft_consensus.cc:2726] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 06:28:18.753366 8581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:18.748161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57254 (local address 127.0.97.67:34719)
0619 06:28:18.748379 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:18.748385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:18.748515 (+ 130us) server_negotiation.cc:407] Connection header received
0619 06:28:18.748673 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:18.748679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:18.748830 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:18.748929 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:18.749307 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.750339 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.751482 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:18.752145 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:18.752267 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:18.752610 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:18.752735 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:18.752831 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:18.752925 (+ 94us) server_negotiation.cc:299] Negotiation successful
0619 06:28:18.753001 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 06:28:18.755909 8293 raft_consensus.cc:1261] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Refusing update from remote peer 5b01b66c1e5f4621bb32fbba97fcf0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:18.756441 8160 raft_consensus.cc:1261] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Refusing update from remote peer 5b01b66c1e5f4621bb32fbba97fcf0a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:18.756425 8539 consensus_queue.cc:1035] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:18.756985 8539 consensus_queue.cc:1035] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:18.767896 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:18.768160 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:18.771198 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Sending DeleteTablet for 3 replicas of tablet 3263dd64f7704710bee8c09cd7947a44
I20260619 06:28:18.772220 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 3263dd64f7704710bee8c09cd7947a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:18 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44212
I20260619 06:28:18.773047 8533 tablet_replica.cc:330] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: stopping tablet replica
I20260619 06:28:18.773265 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet 3263dd64f7704710bee8c09cd7947a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:18 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45030
I20260619 06:28:18.773450 8533 raft_consensus.cc:2229] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:18.773427 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet 3263dd64f7704710bee8c09cd7947a44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:18 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:56178
I20260619 06:28:18.773520 8535 tablet_replica.cc:330] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: stopping tablet replica
I20260619 06:28:18.773610 8533 pending_rounds.cc:62] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Trying to abort 2 pending ops.
I20260619 06:28:18.773672 8534 tablet_replica.cc:330] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: stopping tablet replica
I20260619 06:28:18.773694 8535 raft_consensus.cc:2229] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:28:18.773818 8534 raft_consensus.cc:2229] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:18.773995 8535 raft_consensus.cc:2258] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:18.774017 8534 raft_consensus.cc:2258] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:18.773720 8533 pending_rounds.cc:68] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298459642898706432 op_type: WRITE_OP write_request { tablet_id: "3263dd64f7704710bee8c09cd7947a44" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\256\305\323\321\006\"8\n\ntest-admin\022*\n 65bbd39e53a54dafb1e10863c889f760\020\001\030\001 \001(\001" signature: "<redacted>""\014\227\022SI\315\250\225#\226k\366T\251\031\216\300`\244\304\357\271@\347\314A)\352p\036\271\006\017v\270\206\234\317\217\275\034\376e\026z\230\377&\034\007\321\364\302\361\211\307>\343\220\247\235\303jw" signing_key_seq_num: 0 } } request_id { client_id: "a363837e44834dd18691d0c6a577c395" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260619 06:28:18.774426 8533 pending_rounds.cc:68] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298459642726682624 op_type: NO_OP noop_request { }
I20260619 06:28:18.774564 8533 raft_consensus.cc:2866] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260619 06:28:18.774715 8533 raft_consensus.cc:2258] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:18.775692 8535 ts_tablet_manager.cc:1905] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:18.776320 7998 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:28:18.776846 7998 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:18.783051 8139 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc2860cdd574c919d34a7570bbdb942 (DEFAULT_TABLE table=test-table [id=8f2847d4f9804a5282b8bcd20867ff88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:18.783468 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc2860cdd574c919d34a7570bbdb942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.783869 8274 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc2860cdd574c919d34a7570bbdb942 (DEFAULT_TABLE table=test-table [id=8f2847d4f9804a5282b8bcd20867ff88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:18.784268 8274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc2860cdd574c919d34a7570bbdb942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.787879 8509 tablet_bootstrap.cc:492] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: Bootstrap starting.
I20260619 06:28:18.788434 8412 tablet_service.cc:1467] Processing CreateTablet for tablet 6fc2860cdd574c919d34a7570bbdb942 (DEFAULT_TABLE table=test-table [id=8f2847d4f9804a5282b8bcd20867ff88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:18.788841 8412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fc2860cdd574c919d34a7570bbdb942. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:18.789388 8509 tablet_bootstrap.cc:654] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.791057 8508 tablet_bootstrap.cc:492] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: Bootstrap starting.
I20260619 06:28:18.792601 8508 tablet_bootstrap.cc:654] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.798184 8535 ts_tablet_manager.cc:1918] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:18.798408 8535 log.cc:1196] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/3263dd64f7704710bee8c09cd7947a44
I20260619 06:28:18.798959 8507 tablet_bootstrap.cc:492] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Bootstrap starting.
I20260619 06:28:18.801043 8507 tablet_bootstrap.cc:654] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:18.802933 8534 ts_tablet_manager.cc:1905] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:18.803246 8535 ts_tablet_manager.cc:1939] T 3263dd64f7704710bee8c09cd7947a44 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting consensus metadata
I20260619 06:28:18.803536 8507 tablet_bootstrap.cc:492] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: No bootstrap required, opened a new log
I20260619 06:28:18.803689 8507 ts_tablet_manager.cc:1397] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260619 06:28:18.804540 7983 catalog_manager.cc:4841] TS 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719): tablet 3263dd64f7704710bee8c09cd7947a44 (table test-table [id=65bbd39e53a54dafb1e10863c889f760]) successfully deleted
I20260619 06:28:18.804463 8507 raft_consensus.cc:348] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.804721 8507 raft_consensus.cc:374] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.804804 8507 raft_consensus.cc:729] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b01b66c1e5f4621bb32fbba97fcf0a8, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.805091 8507 consensus_queue.cc:260] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.805612 8507 ts_tablet_manager.cc:1428] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 06:28:18.806566 8509 tablet_bootstrap.cc:492] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: No bootstrap required, opened a new log
I20260619 06:28:18.806763 8509 ts_tablet_manager.cc:1397] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.002s
I20260619 06:28:18.807617 8509 raft_consensus.cc:348] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.807904 8509 raft_consensus.cc:374] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.808017 8509 raft_consensus.cc:729] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92372700c0f4e13901a604976d34df5, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.808269 8509 consensus_queue.cc:260] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.808837 8509 ts_tablet_manager.cc:1428] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260619 06:28:18.812494 8533 ts_tablet_manager.cc:1905] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:18.817497 8534 ts_tablet_manager.cc:1918] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:18.817618 8534 log.cc:1196] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/3263dd64f7704710bee8c09cd7947a44
I20260619 06:28:18.818054 8534 ts_tablet_manager.cc:1939] T 3263dd64f7704710bee8c09cd7947a44 P d92372700c0f4e13901a604976d34df5: Deleting consensus metadata
I20260619 06:28:18.819012 7985 catalog_manager.cc:4841] TS d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261): tablet 3263dd64f7704710bee8c09cd7947a44 (table test-table [id=65bbd39e53a54dafb1e10863c889f760]) successfully deleted
I20260619 06:28:18.822903 8533 ts_tablet_manager.cc:1918] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:18.823062 8533 log.cc:1196] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/3263dd64f7704710bee8c09cd7947a44
I20260619 06:28:18.823545 8533 ts_tablet_manager.cc:1939] T 3263dd64f7704710bee8c09cd7947a44 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting consensus metadata
I20260619 06:28:18.824856 7984 catalog_manager.cc:4841] TS 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293): tablet 3263dd64f7704710bee8c09cd7947a44 (table test-table [id=65bbd39e53a54dafb1e10863c889f760]) successfully deleted
I20260619 06:28:18.829607 8508 tablet_bootstrap.cc:492] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: No bootstrap required, opened a new log
I20260619 06:28:18.829780 8508 ts_tablet_manager.cc:1397] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.024s
I20260619 06:28:18.830375 8508 raft_consensus.cc:348] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.830567 8508 raft_consensus.cc:374] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:18.830641 8508 raft_consensus.cc:729] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326c5bf6dee74c8b89088a5b749c7bba, State: Initialized, Role: FOLLOWER
I20260619 06:28:18.830880 8508 consensus_queue.cc:260] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:18.831301 8508 ts_tablet_manager.cc:1428] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 06:28:19.162904 8515 raft_consensus.cc:482] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:19.163080 8515 raft_consensus.cc:504] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:19.163709 8515 leader_election.cc:288] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293), 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719)
I20260619 06:28:19.164183 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc2860cdd574c919d34a7570bbdb942" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba" is_pre_election: true
I20260619 06:28:19.164214 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc2860cdd574c919d34a7570bbdb942" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" is_pre_election: true
I20260619 06:28:19.164443 8437 raft_consensus.cc:2454] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92372700c0f4e13901a604976d34df5 in term 0.
I20260619 06:28:19.164434 8293 raft_consensus.cc:2454] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92372700c0f4e13901a604976d34df5 in term 0.
I20260619 06:28:19.164924 8087 leader_election.cc:302] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 326c5bf6dee74c8b89088a5b749c7bba, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:19.165205 8515 raft_consensus.cc:2781] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:19.165326 8515 raft_consensus.cc:482] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:19.165414 8515 raft_consensus.cc:3037] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:19.167305 8515 raft_consensus.cc:504] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:19.167884 8515 leader_election.cc:288] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 election: Requested vote from peers 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293), 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719)
I20260619 06:28:19.168226 8293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc2860cdd574c919d34a7570bbdb942" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "326c5bf6dee74c8b89088a5b749c7bba"
I20260619 06:28:19.168344 8437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fc2860cdd574c919d34a7570bbdb942" candidate_uuid: "d92372700c0f4e13901a604976d34df5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8"
I20260619 06:28:19.168437 8293 raft_consensus.cc:3037] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:19.168514 8437 raft_consensus.cc:3037] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:19.171363 8293 raft_consensus.cc:2454] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92372700c0f4e13901a604976d34df5 in term 1.
I20260619 06:28:19.171378 8437 raft_consensus.cc:2454] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92372700c0f4e13901a604976d34df5 in term 1.
I20260619 06:28:19.171762 8087 leader_election.cc:302] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 326c5bf6dee74c8b89088a5b749c7bba, 5b01b66c1e5f4621bb32fbba97fcf0a8, d92372700c0f4e13901a604976d34df5; no voters:
I20260619 06:28:19.172037 8515 raft_consensus.cc:2781] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:19.172364 8515 raft_consensus.cc:686] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 1 LEADER]: Becoming Leader. State: Replica: d92372700c0f4e13901a604976d34df5, State: Running, Role: LEADER
I20260619 06:28:19.172641 8515 consensus_queue.cc:237] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } }
I20260619 06:28:19.175001 7998 catalog_manager.cc:5495] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d92372700c0f4e13901a604976d34df5 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "d92372700c0f4e13901a604976d34df5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92372700c0f4e13901a604976d34df5" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 } health_report { overall_health: UNKNOWN } } }
I20260619 06:28:19.265141 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.260019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.0.97.65:34261)
0619 06:28:19.260463 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:19.260470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:19.260543 (+ 73us) server_negotiation.cc:407] Connection header received
0619 06:28:19.260739 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:19.260746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:19.260839 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:19.260944 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:19.261337 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.262031 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.263359 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.263986 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.264099 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.264442 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:19.264557 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:19.264643 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:19.264752 (+ 109us) server_negotiation.cc:299] Negotiation successful
0619 06:28:19.264826 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":237,"threads_started":1}
I20260619 06:28:19.284157 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.278396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57503 (local address 127.0.97.67:34719)
0619 06:28:19.279151 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:19.279155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:19.279171 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:28:19.279219 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:19.279223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:19.279280 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:19.279368 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:19.279864 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.280599 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.282554 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.282977 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.283066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.283543 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:19.283661 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:19.283728 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:19.283889 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 06:28:19.283946 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":518,"threads_started":1}
I20260619 06:28:19.284353 8599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.278211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:34719 (local address 127.0.97.65:57503)
0619 06:28:19.278848 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:19.278866 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:19.278980 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:19.279402 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:19.279406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:19.279417 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:28:19.279721 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:19.279731 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.280717 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.280723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:19.282430 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:19.282444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.283125 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.283131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:19.283319 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.283381 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:28:19.283755 (+ 374us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:28:19.283768 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 06:28:19.283855 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:28:19.283989 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":338,"threads_started":1}
I20260619 06:28:19.292405 8437 raft_consensus.cc:1261] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Refusing update from remote peer d92372700c0f4e13901a604976d34df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:19.292552 8293 raft_consensus.cc:1261] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Refusing update from remote peer d92372700c0f4e13901a604976d34df5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:19.293100 8591 consensus_queue.cc:1035] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "326c5bf6dee74c8b89088a5b749c7bba" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:19.293655 8591 consensus_queue.cc:1035] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b01b66c1e5f4621bb32fbba97fcf0a8" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 34719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:19.324990 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.319779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:53991 (local address 127.0.97.65:34261)
0619 06:28:19.319938 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:19.319944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:19.319961 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:28:19.320047 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:19.320052 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:19.320140 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:19.320255 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:19.320592 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.321562 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.323388 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.323784 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.323911 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.324287 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:19.324389 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:19.324481 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:19.324613 (+ 132us) server_negotiation.cc:299] Negotiation successful
0619 06:28:19.324687 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:28:19.325070 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.319390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34261 (local address 127.0.97.67:53991)
0619 06:28:19.319789 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:19.319825 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:19.319942 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:19.320271 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:19.320275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:19.320287 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 06:28:19.320460 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:19.320468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.321735 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.321740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:19.323249 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:19.323260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.323929 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.323937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:19.324094 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.324128 (+ 34us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 06:28:19.324481 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 06:28:19.324494 (+ 13us) client_negotiation.cc:769] Sending connection context
0619 06:28:19.324594 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 06:28:19.324719 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":436,"threads_started":1}
I20260619 06:28:19.351809 7998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:19.352061 7998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:19.354565 7998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c91da951f8b44c4bf0c5edd1fcdc93a: Sending DeleteTablet for 3 replicas of tablet 6fc2860cdd574c919d34a7570bbdb942
I20260619 06:28:19.355254 8139 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc2860cdd574c919d34a7570bbdb942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:19 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:56178
I20260619 06:28:19.355271 8274 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc2860cdd574c919d34a7570bbdb942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:19 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44212
I20260619 06:28:19.355398 8412 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fc2860cdd574c919d34a7570bbdb942 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:19 UTC) from {username='oryx', principal='oryx/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45030
I20260619 06:28:19.357110 8609 tablet_replica.cc:330] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: stopping tablet replica
I20260619 06:28:19.357282 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 8069
I20260619 06:28:19.357318 8609 raft_consensus.cc:2229] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:19.357579 8609 raft_consensus.cc:2258] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:19.358577 8607 tablet_replica.cc:330] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: stopping tablet replica
I20260619 06:28:19.358588 8609 ts_tablet_manager.cc:1905] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:19.358846 8607 raft_consensus.cc:2229] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:28:19.359215 8607 raft_consensus.cc:2258] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:19.360148 8607 ts_tablet_manager.cc:1905] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:19.365238 8609 ts_tablet_manager.cc:1918] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:28:19.365408 8609 log.cc:1196] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/6fc2860cdd574c919d34a7570bbdb942
I20260619 06:28:19.365878 8609 ts_tablet_manager.cc:1939] T 6fc2860cdd574c919d34a7570bbdb942 P 5b01b66c1e5f4621bb32fbba97fcf0a8: Deleting consensus metadata
I20260619 06:28:19.366863 8607 ts_tablet_manager.cc:1918] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:28:19.366878 7983 catalog_manager.cc:4841] TS 5b01b66c1e5f4621bb32fbba97fcf0a8 (127.0.97.67:34719): tablet 6fc2860cdd574c919d34a7570bbdb942 (table test-table [id=8f2847d4f9804a5282b8bcd20867ff88]) successfully deleted
I20260619 06:28:19.366987 8607 log.cc:1196] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/6fc2860cdd574c919d34a7570bbdb942
I20260619 06:28:19.367379 8607 ts_tablet_manager.cc:1939] T 6fc2860cdd574c919d34a7570bbdb942 P d92372700c0f4e13901a604976d34df5: Deleting consensus metadata
I20260619 06:28:19.368237 7985 catalog_manager.cc:4841] TS d92372700c0f4e13901a604976d34df5 (127.0.97.65:34261): tablet 6fc2860cdd574c919d34a7570bbdb942 (table test-table [id=8f2847d4f9804a5282b8bcd20867ff88]) successfully deleted
I20260619 06:28:19.368886 8608 tablet_replica.cc:330] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: stopping tablet replica
I20260619 06:28:19.369120 8608 raft_consensus.cc:2229] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:19.369343 8608 raft_consensus.cc:2258] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:19.370154 8608 ts_tablet_manager.cc:1905] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:19.376178 8608 ts_tablet_manager.cc:1918] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 06:28:19.376315 8608 log.cc:1196] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/6fc2860cdd574c919d34a7570bbdb942
I20260619 06:28:19.376716 8608 ts_tablet_manager.cc:1939] T 6fc2860cdd574c919d34a7570bbdb942 P 326c5bf6dee74c8b89088a5b749c7bba: Deleting consensus metadata
Jun 19 06:28:19 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.65@KRBTEST.COM
I20260619 06:28:19.377631 7984 catalog_manager.cc:4841] TS 326c5bf6dee74c8b89088a5b749c7bba (127.0.97.66:39293): tablet 6fc2860cdd574c919d34a7570bbdb942 (table test-table [id=8f2847d4f9804a5282b8bcd20867ff88]) successfully deleted
I20260619 06:28:19.382454 8596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.359533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39294 (local address 127.0.97.65:34261)
0619 06:28:19.359787 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:19.359796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:19.368479 (+ 8683us) server_negotiation.cc:407] Connection header received
0619 06:28:19.368617 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:19.368625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:19.368713 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:19.368821 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:19.370386 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.371060 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.372821 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.373248 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.373338 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.377951 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:19.377962 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:19.377976 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:19.378007 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:19.380376 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:19.380927 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:19.380933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:19.380935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:19.381059 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:19.381407 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:19.381410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:19.381412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:19.381698 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:19.381903 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:19.382069 (+ 166us) server_negotiation.cc:299] Negotiation successful
0619 06:28:19.382234 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260619 06:28:19.383059 8200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:19.504289 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 8069
I20260619 06:28:19.526700 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 8209
Jun 19 06:28:19 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.66@KRBTEST.COM
I20260619 06:28:19.541016 8612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.527384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.0.97.66:39293)
0619 06:28:19.527978 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:19.527982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:19.527995 (+ 13us) server_negotiation.cc:407] Connection header received
0619 06:28:19.528043 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:19.528046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:19.528100 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:19.528218 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:19.529491 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.530205 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.531396 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.531838 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.531983 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.535910 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:19.535921 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:19.535934 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:19.535962 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:19.538460 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:19.539080 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:19.539087 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:19.539089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:19.539207 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:19.539570 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:19.539576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:19.539579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:19.539919 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:19.540191 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:19.540459 (+ 268us) server_negotiation.cc:299] Negotiation successful
0619 06:28:19.540672 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":255,"threads_started":1}
I20260619 06:28:19.541889 8338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:19.636085 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 8209
I20260619 06:28:19.654201 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 8348
Jun 19 06:28:19 dist-test-slave-4t1f krb5kdc[7953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.67@KRBTEST.COM
I20260619 06:28:19.668248 8600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:19.654947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.0.97.67:34719)
0619 06:28:19.655154 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:19.655159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:19.655192 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:28:19.655353 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:19.655357 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:19.655425 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:19.655505 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:19.656669 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.657421 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.658568 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:19.658946 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:19.659028 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:19.663189 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:19.663205 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:19.663221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:19.663270 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:19.665985 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:19.666549 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:19.666556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:19.666558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:19.666675 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:19.666997 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:19.667000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:19.667002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:19.667301 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:19.667513 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:19.667743 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 06:28:19.667908 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260619 06:28:19.668841 8478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:19.811908 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 8348
I20260619 06:28:19.836714 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 7966
I20260619 06:28:19.837944 8038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:19.948566 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 7966
2026-06-19T06:28:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5974 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:28:20 dist-test-slave-4t1f krb5kdc[8619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:28:20 dist-test-slave-4t1f krb5kdc[8619](info): set up 2 sockets
Jun 19 06:28:20 dist-test-slave-4t1f krb5kdc[8619](info): commencing operation
krb5kdc: starting...
W20260619 06:28:21.050032 389 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:28:21 dist-test-slave-4t1f krb5kdc[8619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:28:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:28:21.254184 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46855
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:21.568886 8632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:21.569216 8632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:21.569389 8632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:21.576459 8632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:21.576598 8632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:21.576694 8632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:21.576787 8632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:21.576865 8632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:21.586540 8632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:21.587798 8632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:21.589247 8632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:21.601223 8640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:21.601279 8638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:21.601351 8637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:21.602010 8632 server_base.cc:1034] running on GCE node
I20260619 06:28:21.602772 8632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:21.604018 8632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:21.605232 8632 hybrid_clock.cc:648] HybridClock initialized: now 1781850501605193 us; error 50 us; skew 500 ppm
I20260619 06:28:21.605562 8632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T06:28:21Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1719 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:28:21 dist-test-slave-4t1f krb5kdc[8646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:28:21 dist-test-slave-4t1f krb5kdc[8646](info): set up 2 sockets
Jun 19 06:28:21 dist-test-slave-4t1f krb5kdc[8646](info): commencing operation
krb5kdc: starting...
W20260619 06:28:22.765965 389 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:28:22 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:28:22.965101 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44509
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:23.282624 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:23.282913 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:23.283041 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:23.290091 8659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:23.290197 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:23.290243 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:23.290306 8659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:23.290349 8659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:23.299902 8659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:23.301126 8659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:23.302387 8659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:23.313691 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:23.314072 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:23.313519 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:23.314721 8659 server_base.cc:1034] running on GCE node
I20260619 06:28:23.315465 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:23.316778 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:23.318006 8659 hybrid_clock.cc:648] HybridClock initialized: now 1781850503317960 us; error 52 us; skew 500 ppm
I20260619 06:28:23.318346 8659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:23 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:23.330542 8659 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:28:23.331960 8659 webserver.cc:460] Webserver started at http://127.0.97.126:35691/ using document root <none> and password file <none>
I20260619 06:28:23.332598 8659 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:23.332698 8659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:23.332947 8659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:23.335844 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "f5b9b9d0aa034f4289ea4a224483a058"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "dea5eece706a11d879942d17d6557a4d"
server_key_iv: "853b1493841ceee4786be84b6fb69d9d"
server_key_version: "encryptionkey@0"
I20260619 06:28:23.336545 8659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "f5b9b9d0aa034f4289ea4a224483a058"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "dea5eece706a11d879942d17d6557a4d"
server_key_iv: "853b1493841ceee4786be84b6fb69d9d"
server_key_version: "encryptionkey@0"
I20260619 06:28:23.341517 8659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260619 06:28:23.344743 8674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:23.346021 8659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:28:23.346186 8659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "f5b9b9d0aa034f4289ea4a224483a058"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "dea5eece706a11d879942d17d6557a4d"
server_key_iv: "853b1493841ceee4786be84b6fb69d9d"
server_key_version: "encryptionkey@0"
I20260619 06:28:23.346393 8659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:23.368126 8659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:23.371851 8659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:23.372139 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:23.389802 8659 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44509
I20260619 06:28:23.389969 8725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44509 every 8 connection(s)
I20260619 06:28:23.391075 8659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:23.394387 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 8659
I20260619 06:28:23.394606 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:28:23.394620 8726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:23.395257 389 external_mini_cluster.cc:1428] Setting key f48fc4e45a403bf253be073dfc7f5067
I20260619 06:28:23.407902 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Bootstrap starting.
Jun 19 06:28:23 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:23.410753 8726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:23.411978 8726 log.cc:824] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:23.415242 8726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: No bootstrap required, opened a new log
I20260619 06:28:23.419361 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:23.397190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35496 (local address 127.0.97.126:44509)
0619 06:28:23.397956 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:23.397971 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:23.398018 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:28:23.400762 (+ 2744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:23.400790 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:23.401277 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:23.401721 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:23.402956 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:23.404913 (+ 1957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:23.406138 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:23.407109 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:23.407280 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:23.411079 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:23.411096 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:23.411117 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:23.411168 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:23.415129 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:23.415809 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:23.415817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:23.415824 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:23.416015 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:23.416345 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:23.416350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:23.416353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:23.417024 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:23.417352 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:23.417673 (+ 321us) server_negotiation.cc:299] Negotiation successful
0619 06:28:23.418106 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":311,"threads_started":1}
I20260619 06:28:23.424265 8726 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:23.424553 8726 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:23.424623 8726 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b9b9d0aa034f4289ea4a224483a058, State: Initialized, Role: FOLLOWER
I20260619 06:28:23.425169 8726 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:23.425371 8726 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:23.425459 8726 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:23.425578 8726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:23.427482 8726 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:23.427980 8726 leader_election.cc:302] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5b9b9d0aa034f4289ea4a224483a058; no voters:
I20260619 06:28:23.428566 8726 leader_election.cc:288] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:28:23.428681 8731 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:23.429339 8731 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 LEADER]: Becoming Leader. State: Replica: f5b9b9d0aa034f4289ea4a224483a058, State: Running, Role: LEADER
I20260619 06:28:23.429890 8731 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:23.430565 8726 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:23.432478 8732 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5b9b9d0aa034f4289ea4a224483a058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } } }
I20260619 06:28:23.432777 8732 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:23.433669 8738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:23.433507 8733 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5b9b9d0aa034f4289ea4a224483a058. Latest consensus state: current_term: 1 leader_uuid: "f5b9b9d0aa034f4289ea4a224483a058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } } }
I20260619 06:28:23.433832 8733 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:23.439107 8738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:23.446456 8738 catalog_manager.cc:1347] Generated new cluster ID: e5087641699b4fc3806a2a2aacd1a689
I20260619 06:28:23.446559 8738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:23.470579 8738 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:28:23.472333 8738 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:23.478860 8738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Generated new TSK 0
I20260619 06:28:23.479768 8738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:28:23.532187 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44509
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:23.848428 8752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:23.848685 8752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:23.848873 8752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:23.856376 8752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:23.856518 8752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:23.856709 8752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:23.866376 8752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:23.867641 8752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:23.869011 8752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:23.882560 8758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:23.882439 8757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:23.882472 8760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:23.883514 8752 server_base.cc:1034] running on GCE node
I20260619 06:28:23.884114 8752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:23.885039 8752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:23.886366 8752 hybrid_clock.cc:648] HybridClock initialized: now 1781850503886315 us; error 51 us; skew 500 ppm
I20260619 06:28:23.886696 8752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:23 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:23.889727 8752 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:28:23.891103 8752 webserver.cc:460] Webserver started at http://127.0.97.65:45841/ using document root <none> and password file <none>
I20260619 06:28:23.891786 8752 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:23.891932 8752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:23.892237 8752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:23.895030 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "a7e489eedcfa4af7a26aa5c6d61c28ea"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "c2855772898225ecba920cbf04925892"
server_key_iv: "f6451331151f5f8f6a9dd9330859681e"
server_key_version: "encryptionkey@0"
I20260619 06:28:23.895761 8752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "a7e489eedcfa4af7a26aa5c6d61c28ea"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "c2855772898225ecba920cbf04925892"
server_key_iv: "f6451331151f5f8f6a9dd9330859681e"
server_key_version: "encryptionkey@0"
I20260619 06:28:23.900451 8752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 06:28:23.903666 8767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:23.904840 8752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:28:23.905015 8752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "a7e489eedcfa4af7a26aa5c6d61c28ea"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "c2855772898225ecba920cbf04925892"
server_key_iv: "f6451331151f5f8f6a9dd9330859681e"
server_key_version: "encryptionkey@0"
I20260619 06:28:23.905267 8752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:23.925768 8752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:23.929525 8752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:23.929790 8752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:23.930759 8752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:23.932174 8752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:23.932277 8752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:23.932415 8752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:23.932485 8752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:23.964227 8752 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42333
I20260619 06:28:23.964313 8879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42333 every 8 connection(s)
I20260619 06:28:23.965546 8752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:23.971501 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 8752
I20260619 06:28:23.971712 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:28:23.972522 389 external_mini_cluster.cc:1428] Setting key e8af7d58a3a80fc690b826952eb872b8
Jun 19 06:28:23 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:23.987329 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:23.967987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:54617 (local address 127.0.97.126:44509)
0619 06:28:23.968413 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:23.968419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:23.969496 (+ 1077us) server_negotiation.cc:407] Connection header received
0619 06:28:23.971300 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:23.971304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:23.971370 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:23.971463 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:23.973536 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:23.974531 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:23.976038 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:23.976801 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:23.976936 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:23.980981 (+ 4045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:23.980986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:23.980989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:23.981019 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:23.984093 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:23.984874 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:23.984882 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:23.984889 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:23.985043 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:23.985451 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:23.985458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:23.985464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:23.985850 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:23.985992 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:23.986618 (+ 626us) server_negotiation.cc:299] Negotiation successful
0619 06:28:23.986876 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":197,"threads_started":1}
I20260619 06:28:23.987917 8883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:23.968578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:54617)
0619 06:28:23.969331 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:23.969372 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:23.970971 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:23.971747 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:23.971759 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:23.972490 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:23.973300 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:23.973332 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:23.974709 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:23.974712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:23.975898 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:23.975910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:23.976985 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:23.976990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:23.977238 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:23.977894 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:23.977913 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:23.980760 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:23.984298 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:23.984303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:23.984314 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:23.984719 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:23.985168 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:23.985172 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:23.985174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:23.985334 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:23.985975 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:23.985980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:23.986273 (+ 293us) client_negotiation.cc:769] Sending connection context
0619 06:28:23.986481 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 06:28:23.986838 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":277,"threads_started":1}
I20260619 06:28:23.989427 8880 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44509
I20260619 06:28:23.989715 8880 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:23.990345 8880 heartbeater.cc:511] Master 127.0.97.126:44509 requested a full tablet report, sending...
I20260619 06:28:23.992249 8691 ts_manager.cc:194] Registered new tserver with Master: a7e489eedcfa4af7a26aa5c6d61c28ea (127.0.97.65:42333)
I20260619 06:28:23.993750 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:54617
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:28:24.020473 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44509
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:24.339762 8886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:24.340027 8886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:24.340209 8886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:24.347128 8886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:24.347252 8886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:24.347450 8886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:24.356889 8886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:24.358187 8886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:24.359495 8886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:24.372539 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:24.372448 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:24.372473 8894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:24.373216 8886 server_base.cc:1034] running on GCE node
I20260619 06:28:24.373764 8886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:24.374737 8886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:24.375918 8886 hybrid_clock.cc:648] HybridClock initialized: now 1781850504375868 us; error 47 us; skew 500 ppm
I20260619 06:28:24.376211 8886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:24 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:24.379132 8886 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:28:24.380514 8886 webserver.cc:460] Webserver started at http://127.0.97.66:38673/ using document root <none> and password file <none>
I20260619 06:28:24.381213 8886 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:24.381359 8886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:24.381654 8886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:24.384357 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "44454405fc7349a9800f228d93ead219"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "b5d42318dd9817449523ccef1d9b620d"
server_key_iv: "ba0b93d8f3b3ed665fb6039c6c9946b2"
server_key_version: "encryptionkey@0"
I20260619 06:28:24.385166 8886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "44454405fc7349a9800f228d93ead219"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "b5d42318dd9817449523ccef1d9b620d"
server_key_iv: "ba0b93d8f3b3ed665fb6039c6c9946b2"
server_key_version: "encryptionkey@0"
I20260619 06:28:24.389922 8886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:28:24.393396 8901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:24.394723 8886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:28:24.394887 8886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "44454405fc7349a9800f228d93ead219"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "b5d42318dd9817449523ccef1d9b620d"
server_key_iv: "ba0b93d8f3b3ed665fb6039c6c9946b2"
server_key_version: "encryptionkey@0"
I20260619 06:28:24.395078 8886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:24.407006 8886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:24.411036 8886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:24.411360 8886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:24.412197 8886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:24.413493 8886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:24.413592 8886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:24.413736 8886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:24.413807 8886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:24.445218 8886 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:45299
I20260619 06:28:24.445297 9013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:45299 every 8 connection(s)
I20260619 06:28:24.446398 8886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:24.448800 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 8886
I20260619 06:28:24.448992 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:28:24.449518 389 external_mini_cluster.cc:1428] Setting key 9ffe0932f7b23d6ebf09e6c537b14827
Jun 19 06:28:24 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:24.468250 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:24.448937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:49415 (local address 127.0.97.126:44509)
0619 06:28:24.449162 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:24.449167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:24.450576 (+ 1409us) server_negotiation.cc:407] Connection header received
0619 06:28:24.452935 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:24.452939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:24.452995 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:24.453132 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:24.455141 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.455825 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.458279 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.458653 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.458752 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:24.462588 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:24.462593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:24.462596 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:24.462624 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:24.465185 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:24.466052 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:24.466056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:24.466058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:24.466179 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:24.466658 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:24.466661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:24.466663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:24.466887 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:24.466998 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:24.467891 (+ 893us) server_negotiation.cc:299] Negotiation successful
0619 06:28:24.468039 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:28:24.469317 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:24.449568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:49415)
0619 06:28:24.450403 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:24.450449 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:24.452656 (+ 2207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:24.453384 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:24.453399 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:24.454185 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:24.454980 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:24.455007 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.455979 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.455988 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:24.458145 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:24.458167 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.458874 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.458883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:24.459125 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:24.459797 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:24.459820 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:24.462370 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:24.465307 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:24.465318 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:24.465336 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:24.465945 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:24.466286 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:24.466293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:24.466300 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:24.466532 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:24.467011 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:24.467022 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:24.467473 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 06:28:24.467786 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 06:28:24.468199 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":302,"threads_started":1}
I20260619 06:28:24.470703 9014 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44509
I20260619 06:28:24.471068 9014 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:24.471931 9014 heartbeater.cc:511] Master 127.0.97.126:44509 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260619 06:28:24.473532 8691 ts_manager.cc:194] Registered new tserver with Master: 44454405fc7349a9800f228d93ead219 (127.0.97.66:45299)
I20260619 06:28:24.474574 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:49415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:28:24.497824 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44509
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:24.812529 9019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:24.812801 9019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:24.812892 9019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:24.819854 9019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:24.819947 9019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:24.820052 9019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:24.829210 9019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:24.830348 9019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:24.831636 9019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:24.844686 9024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:24.844650 9025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:24.844726 9027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:24.845623 9019 server_base.cc:1034] running on GCE node
I20260619 06:28:24.846174 9019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:24.847064 9019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:24.848276 9019 hybrid_clock.cc:648] HybridClock initialized: now 1781850504848225 us; error 61 us; skew 500 ppm
I20260619 06:28:24.848588 9019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:24 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:24.851899 9019 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:28:24.853338 9019 webserver.cc:460] Webserver started at http://127.0.97.67:42049/ using document root <none> and password file <none>
I20260619 06:28:24.853971 9019 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:24.854112 9019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:24.854441 9019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:24.857194 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "b7402d1933fd4902ba9dbfce19ef644b"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "a04565bc4e350088724f16fb8f0b6e3f"
server_key_iv: "d95da3208a532b2c899b5ba5d6e5adf6"
server_key_version: "encryptionkey@0"
I20260619 06:28:24.857954 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "b7402d1933fd4902ba9dbfce19ef644b"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "a04565bc4e350088724f16fb8f0b6e3f"
server_key_iv: "d95da3208a532b2c899b5ba5d6e5adf6"
server_key_version: "encryptionkey@0"
I20260619 06:28:24.862715 9019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:28:24.865912 9034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:24.867064 9019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:28:24.867229 9019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "b7402d1933fd4902ba9dbfce19ef644b"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "a04565bc4e350088724f16fb8f0b6e3f"
server_key_iv: "d95da3208a532b2c899b5ba5d6e5adf6"
server_key_version: "encryptionkey@0"
I20260619 06:28:24.867441 9019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:24.880506 9019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:24.884660 9019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:24.884954 9019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:24.885895 9019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:24.887144 9019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:24.887246 9019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:24.887405 9019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:24.887481 9019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:24.919801 9019 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:45457
I20260619 06:28:24.919842 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:45457 every 8 connection(s)
I20260619 06:28:24.921036 9019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:24.926424 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9019
I20260619 06:28:24.926615 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:28:24.927035 389 external_mini_cluster.cc:1428] Setting key 8a6f4f96641f2aa258653cd1a5214415
Jun 19 06:28:24 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:24.939878 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:24.923382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34295 (local address 127.0.97.126:44509)
0619 06:28:24.923581 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:24.923588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:24.924679 (+ 1091us) server_negotiation.cc:407] Connection header received
0619 06:28:24.926445 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:24.926450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:24.926557 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:24.926665 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:24.928329 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.929310 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.931150 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.931558 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.931649 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:24.934871 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:24.934875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:24.934879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:24.934909 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:24.937570 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:24.938174 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:24.938178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:24.938180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:24.938296 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:24.938623 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:24.938626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:24.938628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:24.938874 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:24.938977 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:24.939517 (+ 540us) server_negotiation.cc:299] Negotiation successful
0619 06:28:24.939655 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:28:24.941290 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:24.923846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:34295)
0619 06:28:24.924516 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:24.924562 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:24.926173 (+ 1611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:24.926899 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:24.926907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:24.927447 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:24.928166 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:24.928193 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.929457 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.929462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:24.930980 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:24.930993 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:24.931740 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:24.931744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:24.931959 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:24.932579 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:24.932600 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:24.934697 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:24.937692 (+ 2995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:24.937705 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:24.937716 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:24.938084 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:24.938381 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:24.938384 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:24.938386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:24.938539 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:24.938971 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:24.938976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:24.939253 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 06:28:24.939448 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 06:28:24.939773 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":269,"threads_started":1}
I20260619 06:28:24.942535 9147 heartbeater.cc:347] Connected to a master server at 127.0.97.126:44509
I20260619 06:28:24.942786 9147 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:24.943384 9147 heartbeater.cc:511] Master 127.0.97.126:44509 requested a full tablet report, sending...
I20260619 06:28:24.944633 8691 ts_manager.cc:194] Registered new tserver with Master: b7402d1933fd4902ba9dbfce19ef644b (127.0.97.67:45457)
I20260619 06:28:24.945431 8691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:34295
I20260619 06:28:24.954052 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab.
Entry for principal oryx/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab.
I20260619 06:28:24.996484 8880 heartbeater.cc:503] Master 127.0.97.126:44509 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab.
Entry for principal oryx/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab.
WARNING: no policy specified for oryx/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab.
Entry for principal oryx/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab.
I20260619 06:28:25.079975 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 8752
Jun 19 06:28:25 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:28:25.096876 9157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:25.081403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.0.97.65:42333)
0619 06:28:25.082108 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:25.082118 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:25.082151 (+ 33us) server_negotiation.cc:407] Connection header received
0619 06:28:25.082336 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:25.082343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:25.082561 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:25.082744 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:25.084021 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:25.085010 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:25.086369 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:25.086982 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:25.087129 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:25.091495 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:25.091506 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:25.091518 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:25.091547 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:25.094226 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:25.094911 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:25.094919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:25.094921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:25.095038 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:25.095396 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:25.095403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:25.095407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:25.095721 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:25.096009 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:25.096292 (+ 283us) server_negotiation.cc:299] Negotiation successful
0619 06:28:25.096541 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":373,"threads_started":1}
I20260619 06:28:25.098134 8875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:25.179388 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 8752
I20260619 06:28:25.195624 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 8886
Jun 19 06:28:25 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:28:25.210927 9159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:25.196704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.0.97.66:45299)
0619 06:28:25.197287 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:25.197294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:25.197321 (+ 27us) server_negotiation.cc:407] Connection header received
0619 06:28:25.197385 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:25.197394 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:25.197648 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:25.197785 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:25.199132 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:25.200147 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:25.201408 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:25.201940 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:25.202041 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:25.205806 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:25.205816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:25.205829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:25.205860 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:25.208623 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:25.209189 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:25.209196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:25.209198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:25.209306 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:25.209589 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:25.209592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:25.209594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:25.209946 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:25.210166 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:25.210370 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 06:28:25.210541 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":255,"threads_started":1}
I20260619 06:28:25.212155 9009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:25.292702 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 8886
I20260619 06:28:25.308338 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9019
Jun 19 06:28:25 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:28:25.323872 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:25.309248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33236 (local address 127.0.97.67:45457)
0619 06:28:25.309805 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:25.309812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:25.309840 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:28:25.309899 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:25.309906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:25.310110 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:25.310284 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:25.311619 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:25.312650 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:25.313926 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:25.314498 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:25.314636 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:25.318825 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:25.318834 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:25.318848 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:25.318878 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:25.321621 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:25.322223 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:25.322230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:25.322232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:25.322333 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:25.322641 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:25.322644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:25.322645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:25.322961 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:25.323175 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:25.323404 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 06:28:25.323584 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":254,"threads_started":1}
I20260619 06:28:25.325181 9142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:25.411016 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9019
I20260619 06:28:25.426462 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 8659
I20260619 06:28:25.427546 8721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:25.528645 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 8659
I20260619 06:28:25.548360 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44509
--webserver_interface=127.0.97.126
--webserver_port=35691
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:25.857895 9164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:25.858157 9164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:25.858263 9164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:25.865655 9164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:25.865749 9164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:25.865795 9164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:25.865845 9164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:25.865901 9164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:25.875025 9164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=35691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:25.876116 9164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:25.877315 9164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:25.888533 9172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:25.888594 9169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:25.888571 9170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:25.889653 9164 server_base.cc:1034] running on GCE node
I20260619 06:28:25.890403 9164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:25.891665 9164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:25.892890 9164 hybrid_clock.cc:648] HybridClock initialized: now 1781850505892845 us; error 61 us; skew 500 ppm
I20260619 06:28:25.893258 9164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:25 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:25.896068 9164 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:28:25.897433 9164 webserver.cc:460] Webserver started at http://127.0.97.126:35691/ using document root <none> and password file <none>
I20260619 06:28:25.898083 9164 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:25.898229 9164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:25.903244 9164 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260619 06:28:25.905831 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:25.907006 9164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260619 06:28:25.907155 9164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "f5b9b9d0aa034f4289ea4a224483a058"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "dea5eece706a11d879942d17d6557a4d"
server_key_iv: "853b1493841ceee4786be84b6fb69d9d"
server_key_version: "encryptionkey@0"
I20260619 06:28:25.908025 9164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:25.939049 9164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:25.942589 9164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:25.942863 9164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:25.960342 9164 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44509
I20260619 06:28:25.960388 9230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44509 every 8 connection(s)
I20260619 06:28:25.961743 9164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:25.968118 9231 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:28:25.969425 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9164
I20260619 06:28:25.970304 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:42333
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:44509
--webserver_port=45841
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:28:25.972314 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Bootstrap starting.
I20260619 06:28:25.992161 9231 log.cc:824] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:26.000610 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:28:26.001170 9231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Bootstrap complete.
I20260619 06:28:26.009848 9231 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:26.010700 9231 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5b9b9d0aa034f4289ea4a224483a058, State: Initialized, Role: FOLLOWER
I20260619 06:28:26.011359 9231 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:26.011570 9231 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:26.011667 9231 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:26.011873 9231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:28:26.015519 9231 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:26.015983 9231 leader_election.cc:302] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5b9b9d0aa034f4289ea4a224483a058; no voters:
I20260619 06:28:26.016602 9231 leader_election.cc:288] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:28:26.016721 9235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:28:26.017935 9235 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [term 2 LEADER]: Becoming Leader. State: Replica: f5b9b9d0aa034f4289ea4a224483a058, State: Running, Role: LEADER
I20260619 06:28:26.018517 9235 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } }
I20260619 06:28:26.018802 9231 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:26.021030 9237 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5b9b9d0aa034f4289ea4a224483a058. Latest consensus state: current_term: 2 leader_uuid: "f5b9b9d0aa034f4289ea4a224483a058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } } }
I20260619 06:28:26.021699 9237 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:26.021318 9236 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f5b9b9d0aa034f4289ea4a224483a058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5b9b9d0aa034f4289ea4a224483a058" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44509 } } }
I20260619 06:28:26.022382 9236 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:26.027477 9241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:26.034247 9241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:26.036051 9241 catalog_manager.cc:1259] Loaded cluster ID: e5087641699b4fc3806a2a2aacd1a689
I20260619 06:28:26.036245 9241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:26.041334 9241 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:26.043115 9241 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f5b9b9d0aa034f4289ea4a224483a058: Loaded TSK: 0
I20260619 06:28:26.045243 9241 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:28:26.327883 9233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:26.328212 9233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:26.328387 9233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:26.335839 9233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:26.335994 9233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:26.336184 9233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:26.345875 9233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.65.keytab
--principal=oryx/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:42333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=45841
--tserver_master_addrs=127.0.97.126:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:26.347196 9233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:26.348431 9233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:26.362962 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:26.362928 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:26.363603 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:26.363948 9233 server_base.cc:1034] running on GCE node
I20260619 06:28:26.364557 9233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:26.365549 9233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:26.366801 9233 hybrid_clock.cc:648] HybridClock initialized: now 1781850506366747 us; error 48 us; skew 500 ppm
I20260619 06:28:26.367084 9233 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850506, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:26.370162 9233 init.cc:377] Logged in from keytab as oryx/127.0.97.65@KRBTEST.COM (short username oryx)
I20260619 06:28:26.371593 9233 webserver.cc:460] Webserver started at http://127.0.97.65:45841/ using document root <none> and password file <none>
I20260619 06:28:26.372315 9233 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:26.372448 9233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:26.378099 9233 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:28:26.380786 9267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:26.382135 9233 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:28:26.382330 9233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "a7e489eedcfa4af7a26aa5c6d61c28ea"
format_stamp: "Formatted at 2026-06-19 06:28:23 on dist-test-slave-4t1f"
server_key: "c2855772898225ecba920cbf04925892"
server_key_iv: "f6451331151f5f8f6a9dd9330859681e"
server_key_version: "encryptionkey@0"
I20260619 06:28:26.383249 9233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:26.411094 9233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:26.415496 9233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:26.415714 9233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:26.416565 9233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:26.417857 9233 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:26.417948 9233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:26.418026 9233 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:26.418079 9233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:26.450621 9233 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42333
I20260619 06:28:26.450706 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42333 every 8 connection(s)
I20260619 06:28:26.451988 9233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:26.453658 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9233
I20260619 06:28:26.454427 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:45299
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:44509
--webserver_port=38673
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:26.469586 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:26.455204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:47079)
0619 06:28:26.455937 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:26.455976 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:26.457891 (+ 1915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:26.459190 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:26.459200 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:26.459776 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:26.460289 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.460305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.461926 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.461937 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.463071 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.463082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.464081 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.464087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.464345 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.465012 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:26.465032 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:26.468687 (+ 3655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":926,"threads_started":1}
W20260619 06:28:26.470237 9380 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:44509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:44509: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
W20260619 06:28:26.471019 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:26.456003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47079 (local address 127.0.97.126:44509)
0619 06:28:26.457519 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:26.457535 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:26.457584 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:28:26.458385 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:26.458398 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:26.458706 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:26.459049 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:26.460474 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.461773 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.463249 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.463865 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.464048 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.469169 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:47079: BlockingRecv error: recv got EOF from 127.0.97.65:47079 (error 108)
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":1114,"threads_started":1}
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:26.478703 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:26.470846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:40935)
0619 06:28:26.471123 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:26.471145 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:26.471267 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:26.472040 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:26.472048 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:26.472344 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:26.472466 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.472473 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.473497 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.473504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.474374 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.474384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.475019 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.475024 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.475191 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.475768 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:26.475783 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:26.478450 (+ 2667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139}
W20260619 06:28:26.479033 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:26.471090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:40935 (local address 127.0.97.126:44509)
0619 06:28:26.471639 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:26.471647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:26.471669 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:28:26.471733 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:26.471740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:26.471837 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:26.472003 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:26.472594 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.473365 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.474508 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.474886 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.475010 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.478689 (+ 3679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:40935: BlockingRecv error: recv got EOF from 127.0.97.65:40935 (error 108)
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":765,"threads_started":1}
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:26.487686 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:26.479321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:58283)
0619 06:28:26.479485 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:26.479503 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:26.479588 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:26.479870 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:26.479875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:26.480250 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:26.480422 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.480432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.481407 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.481412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.483000 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.483015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.483620 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.483626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.483821 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.484350 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:26.484372 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:26.487424 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:28:26.487879 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:26.479388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:58283 (local address 127.0.97.126:44509)
0619 06:28:26.479531 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:26.479534 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:26.479548 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:28:26.479684 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:26.479687 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:26.479734 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:26.479823 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:26.480547 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.481286 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.483124 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.483493 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.483580 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.487628 (+ 4048us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:58283: BlockingRecv error: recv got EOF from 127.0.97.65:58283 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260619 06:28:26.488132 9380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:28:26.772178 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:26.772435 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:26.772527 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:26.779560 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:26.779652 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:26.779769 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:26.788954 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.66.keytab
--principal=oryx/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:45299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=38673
--tserver_master_addrs=127.0.97.126:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:26.790136 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:26.791371 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:26.803821 9393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:26.803959 9391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:26.803925 9390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:26.805698 9384 server_base.cc:1034] running on GCE node
I20260619 06:28:26.806273 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:26.807128 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:26.808321 9384 hybrid_clock.cc:648] HybridClock initialized: now 1781850506808254 us; error 59 us; skew 500 ppm
I20260619 06:28:26.808609 9384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850506, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:26.811645 9384 init.cc:377] Logged in from keytab as oryx/127.0.97.66@KRBTEST.COM (short username oryx)
I20260619 06:28:26.812999 9384 webserver.cc:460] Webserver started at http://127.0.97.66:38673/ using document root <none> and password file <none>
I20260619 06:28:26.813766 9384 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:26.813925 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:26.818909 9384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 06:28:26.821532 9400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:26.822741 9384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260619 06:28:26.822899 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "44454405fc7349a9800f228d93ead219"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "b5d42318dd9817449523ccef1d9b620d"
server_key_iv: "ba0b93d8f3b3ed665fb6039c6c9946b2"
server_key_version: "encryptionkey@0"
I20260619 06:28:26.823850 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:26.838397 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:26.842191 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:26.842450 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:26.843288 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:26.844554 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:26.844668 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:26.844797 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:26.844905 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:26.876427 9384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:45299
I20260619 06:28:26.876518 9512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:45299 every 8 connection(s)
I20260619 06:28:26.877817 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:26.884707 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9384
I20260619 06:28:26.885290 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:45457
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:44509
--webserver_port=42049
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:28:26.893347 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:26.880423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34879 (local address 127.0.97.126:44509)
0619 06:28:26.880670 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:26.880678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:26.881873 (+ 1195us) server_negotiation.cc:407] Connection header received
0619 06:28:26.883628 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:26.883633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:26.883714 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:26.883834 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:26.885343 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.886226 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.887822 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.888225 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.888350 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.893096 (+ 4746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:34879: BlockingRecv error: recv got EOF from 127.0.97.66:34879 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 06:28:26.893985 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:26.880892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:34879)
0619 06:28:26.881739 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:26.881774 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:26.883378 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:26.884058 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:26.884066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:26.884539 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:26.885184 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.885201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.886376 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.886382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.887679 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.887691 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.888443 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.888450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.888675 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.889351 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:26.889370 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:26.892910 (+ 3540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":431,"threads_started":1}
W20260619 06:28:26.894666 9513 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:44509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:44509: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:26.904018 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:26.895192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:39533)
0619 06:28:26.895388 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:26.895408 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:26.895503 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:26.895771 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:26.895778 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:26.896210 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:26.896413 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.896426 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.897387 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.897395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.899093 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.899109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.899792 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.899801 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.900016 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.900603 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:26.900633 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:26.903741 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260619 06:28:26.904157 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:26.895379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39533 (local address 127.0.97.126:44509)
0619 06:28:26.895527 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:26.895531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:26.895548 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:28:26.895585 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:26.895588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:26.895635 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:26.895719 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:26.896543 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.897261 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.899245 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.899625 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.899739 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.903930 (+ 4191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:39533: BlockingRecv error: recv got EOF from 127.0.97.66:39533 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:26 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:26.913482 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:26.904767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:35049)
0619 06:28:26.904990 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:26.905012 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:26.905164 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:26.905461 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:26.905467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:26.905872 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:26.906085 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.906100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.907009 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.907016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.908625 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:26.908638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.909359 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.909365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:26.909557 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.910078 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:26.910099 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:26.913198 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:28:26.913919 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:26.904890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35049 (local address 127.0.97.126:44509)
0619 06:28:26.905022 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:26.905025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:26.905107 (+ 82us) server_negotiation.cc:407] Connection header received
0619 06:28:26.905275 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:26.905279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:26.905328 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:26.905402 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:26.906238 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.906874 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.908758 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:26.909237 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:26.909326 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:26.913583 (+ 4257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:35049: BlockingRecv error: recv got EOF from 127.0.97.66:35049 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 06:28:26.914338 9513 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:28:27.211859 9516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:27.212165 9516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:27.212330 9516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:27.219426 9516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:27.219555 9516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:27.219732 9516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:27.229246 9516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/krb5kdc/oryx_127.0.97.67.keytab
--principal=oryx/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:45457
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=42049
--tserver_master_addrs=127.0.97.126:44509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:27.230551 9516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:27.231817 9516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:27.244683 9521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:27.244755 9522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:27.244719 9524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:27.245426 9516 server_base.cc:1034] running on GCE node
I20260619 06:28:27.245970 9516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:27.246860 9516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:27.248075 9516 hybrid_clock.cc:648] HybridClock initialized: now 1781850507248037 us; error 49 us; skew 500 ppm
I20260619 06:28:27.248366 9516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850507, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:27.251261 9516 init.cc:377] Logged in from keytab as oryx/127.0.97.67@KRBTEST.COM (short username oryx)
I20260619 06:28:27.252568 9516 webserver.cc:460] Webserver started at http://127.0.97.67:42049/ using document root <none> and password file <none>
I20260619 06:28:27.253315 9516 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:27.253458 9516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:27.258374 9516 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:28:27.260958 9531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:27.262293 9516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:28:27.262467 9516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "b7402d1933fd4902ba9dbfce19ef644b"
format_stamp: "Formatted at 2026-06-19 06:28:24 on dist-test-slave-4t1f"
server_key: "a04565bc4e350088724f16fb8f0b6e3f"
server_key_iv: "d95da3208a532b2c899b5ba5d6e5adf6"
server_key_version: "encryptionkey@0"
I20260619 06:28:27.263374 9516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:27.275068 9516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:27.278692 9516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:27.279009 9516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:27.279961 9516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:27.281406 9516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:27.281523 9516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:27.281656 9516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:27.281744 9516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:27.313611 9516 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:45457
I20260619 06:28:27.313694 9643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:45457 every 8 connection(s)
I20260619 06:28:27.315029 9516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:27.316620 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9516
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
W20260619 06:28:27.336731 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:27.317684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:33351 (local address 127.0.97.126:44509)
0619 06:28:27.325385 (+ 7701us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:27.325389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:27.325405 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:28:27.325455 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:27.325459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:27.325516 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:27.325605 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:27.327328 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.328537 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.330502 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.331143 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.331242 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.336443 (+ 5201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:33351: BlockingRecv error: recv got EOF from 127.0.97.67:33351 (error 108)
Metrics: {"server-negotiator.queue_time_us":7612,"thread_start_us":7532,"threads_started":1}
I20260619 06:28:27.337397 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:27.318162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:33351)
0619 06:28:27.318998 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:27.319042 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:27.320787 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:27.325948 (+ 5161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:27.325959 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:27.326492 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:27.327110 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.327135 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.328684 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.328689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.330369 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.330380 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.331345 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.331352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.331638 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.332427 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:27.332459 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:27.336194 (+ 3735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":390,"threads_started":1}
W20260619 06:28:27.338119 9644 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:44509 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.126:44509: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
I20260619 06:28:27.341251 9383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:27.317574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35504 (local address 127.0.97.126:44509)
0619 06:28:27.317767 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:27.317773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:27.326178 (+ 8405us) server_negotiation.cc:407] Connection header received
0619 06:28:27.326367 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:27.326374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:27.326431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:27.326531 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:27.328438 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.329517 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.331023 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.331640 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.331757 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.334476 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:27.334490 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:27.334514 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:27.334564 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:27.337711 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:27.338524 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:27.338534 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:27.338545 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:27.338745 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:27.339422 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:27.339426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:27.339428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:27.339847 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:27.340165 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:27.340525 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 06:28:27.340886 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:27.346633 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:27.338728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:47323)
0619 06:28:27.338913 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:27.338936 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:27.339010 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:27.339265 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:27.339270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:27.339612 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:27.339750 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.339758 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.341141 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.341145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.342096 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.342108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.343033 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.343037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.343159 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.343673 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:27.343689 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:27.346374 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260619 06:28:27.346876 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:27.338825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:47323 (local address 127.0.97.126:44509)
0619 06:28:27.338962 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:27.338966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:27.338978 (+ 12us) server_negotiation.cc:407] Connection header received
0619 06:28:27.339080 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:27.339083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:27.339136 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:27.339204 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:27.339878 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.340946 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.342227 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.342846 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.342956 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.346542 (+ 3586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:47323: BlockingRecv error: recv got EOF from 127.0.97.67:47323 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:27.354149 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:27.347226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:53179)
0619 06:28:27.347382 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:27.347395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:27.347499 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:27.347829 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:27.347834 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:27.348103 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:27.348232 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.348239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.349171 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.349178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.350058 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.350068 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.350803 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.350808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.350987 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.351502 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:27.351517 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:27.353865 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:28:27.354288 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:27.347345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:53179 (local address 127.0.97.126:44509)
0619 06:28:27.347560 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:27.347565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:27.347582 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:28:27.347625 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:27.347628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:27.347683 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:27.347806 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:27.348360 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.349011 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.350251 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.350666 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.350795 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.354025 (+ 3230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:53179: BlockingRecv error: recv got EOF from 127.0.97.67:53179 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260619 06:28:27.354537 9644 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:27.497574 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:27.488731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:54527)
0619 06:28:27.489374 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:27.489393 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:27.489517 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:27.489831 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:27.489834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:27.490325 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:27.490532 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.490541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.491487 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.491491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.492478 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.492491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.493241 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.493246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.493397 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.494078 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:27.494096 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:27.497252 (+ 3156us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":390,"threads_started":1}
W20260619 06:28:27.497714 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:27.488771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:54527 (local address 127.0.97.126:44509)
0619 06:28:27.488930 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:27.488955 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:27.489472 (+ 517us) server_negotiation.cc:407] Connection header received
0619 06:28:27.489635 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:27.489639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:27.489705 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:27.489781 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:27.490675 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.491342 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.492603 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.493013 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.493116 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.497457 (+ 4341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:54527: BlockingRecv error: recv got EOF from 127.0.97.65:54527 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:27 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:27.923147 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:27.914852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:58937)
0619 06:28:27.915287 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:27.915302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:27.915408 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:27.915682 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:27.915685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:27.916111 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:27.916300 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.916309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.917294 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.917298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.918388 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:27.918402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.919436 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.919442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:27.919573 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.920278 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:27.920294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:27.922918 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":234,"threads_started":1}
W20260619 06:28:27.923319 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:27.914985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58937 (local address 127.0.97.126:44509)
0619 06:28:27.915192 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:27.915198 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:27.915377 (+ 179us) server_negotiation.cc:407] Connection header received
0619 06:28:27.915503 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:27.915506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:27.915563 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:27.915647 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:27.916411 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.917173 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.918554 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:27.919220 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:27.919375 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:27.923096 (+ 3721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:58937: BlockingRecv error: recv got EOF from 127.0.97.66:58937 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Jun 19 06:28:28 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:28 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:28.365731 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:28.355101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:56053)
0619 06:28:28.355576 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:28.355591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:28.355747 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:28.356069 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:28.356073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:28.356539 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:28.356724 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:28.356732 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.357760 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.357766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:28.358698 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:28.358709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.359443 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.359450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:28.359632 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:28.362818 (+ 3186us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:28.362836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:28.365504 (+ 2668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
W20260619 06:28:28.365923 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:28.355155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56053 (local address 127.0.97.126:44509)
0619 06:28:28.355358 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:28.355366 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:28.355685 (+ 319us) server_negotiation.cc:407] Connection header received
0619 06:28:28.355873 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:28.355879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:28.355942 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:28.356039 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:28.356861 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.357627 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.358833 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.359276 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.359409 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:28.365683 (+ 6274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:56053: BlockingRecv error: recv got EOF from 127.0.97.67:56053 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:28:28 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:28 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:28.509852 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:28.498560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:37627)
0619 06:28:28.499044 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:28.499059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:28.499215 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:28.499526 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:28.499530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:28.499963 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:28.500102 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:28.500111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.501184 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.501189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:28.502348 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:28.502361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.503064 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.503069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:28.503225 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:28.506733 (+ 3508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:28.506750 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:28.509478 (+ 2728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
W20260619 06:28:28.509935 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:28.498613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:37627 (local address 127.0.97.126:44509)
0619 06:28:28.498822 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:28.498830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:28.499147 (+ 317us) server_negotiation.cc:407] Connection header received
0619 06:28:28.499329 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:28.499336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:28.499399 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:28.499500 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:28.500231 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.500973 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.502492 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.502914 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.503048 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:28.509697 (+ 6649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:37627: BlockingRecv error: recv got EOF from 127.0.97.65:37627 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 06:28:28 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:28 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:28.932157 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:28.924068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:34821)
0619 06:28:28.924598 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:28.924613 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:28.924733 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:28.925013 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:28.925016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:28.925393 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:28.925528 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:28.925537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.926453 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.926456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:28.927485 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:28.927497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.928213 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.928217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:28.928317 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:28.928838 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:28.928853 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:28.931792 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":278,"threads_started":1}
W20260619 06:28:28.932241 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:28.924126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34821 (local address 127.0.97.126:44509)
0619 06:28:28.924319 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:28.924324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:28.924693 (+ 369us) server_negotiation.cc:407] Connection header received
0619 06:28:28.924832 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:28.924835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:28.924895 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:28.924973 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:28.925665 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.926341 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.927618 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:28.928094 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:28.928185 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:28.931991 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:34821: BlockingRecv error: recv got EOF from 127.0.97.66:34821 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 19 06:28:29 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:29 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:29.375185 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:29.366748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:54829)
0619 06:28:29.367501 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:29.367527 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:29.367683 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:29.368032 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:29.368037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:29.368457 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:29.368665 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:29.368675 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.369716 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.369724 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:29.370617 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:29.370628 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.371297 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.371303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:29.371464 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:29.372106 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:29.372123 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:29.374897 (+ 2774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":513,"threads_started":1}
W20260619 06:28:29.375262 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:29.366890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54829 (local address 127.0.97.126:44509)
0619 06:28:29.367063 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:29.367069 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:29.367621 (+ 552us) server_negotiation.cc:407] Connection header received
0619 06:28:29.367811 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:29.367816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:29.367900 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:29.368000 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:29.368805 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.369578 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.370760 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.371148 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.371290 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:29.375086 (+ 3796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:54829: BlockingRecv error: recv got EOF from 127.0.97.67:54829 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 06:28:29 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:29 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:29.518520 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:29.510821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:47863)
0619 06:28:29.511327 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:29.511344 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:29.511452 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:29.511720 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:29.511723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:29.512164 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:29.512338 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:29.512347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.513298 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.513301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:29.514179 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:29.514190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.514763 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.514767 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:29.514880 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:29.515538 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:29.515555 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:29.518249 (+ 2694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
W20260619 06:28:29.518620 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:29.510838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47863 (local address 127.0.97.126:44509)
0619 06:28:29.511061 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:29.511080 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:29.511419 (+ 339us) server_negotiation.cc:407] Connection header received
0619 06:28:29.511541 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:29.511545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:29.511604 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:29.511679 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:29.512460 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.513194 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.514292 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.514652 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.514736 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:29.518452 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:47863: BlockingRecv error: recv got EOF from 127.0.97.65:47863 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:28:29 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:29 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:29.941020 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:29.933022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:40945)
0619 06:28:29.933520 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:29.933535 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:29.933699 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:29.934016 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:29.934021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:29.934378 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:29.934539 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:29.934547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.935574 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.935585 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:29.936515 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:29.936526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.937251 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.937256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:29.937433 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:29.937992 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:29.938008 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:29.940681 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":264,"threads_started":1}
W20260619 06:28:29.941102 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:29.933113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:40945 (local address 127.0.97.126:44509)
0619 06:28:29.933333 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:29.933341 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:29.933631 (+ 290us) server_negotiation.cc:407] Connection header received
0619 06:28:29.933815 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:29.933821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:29.933887 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:29.933987 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:29.934691 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.935422 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.936655 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:29.937112 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:29.937243 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:29.940848 (+ 3605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:40945: BlockingRecv error: recv got EOF from 127.0.97.66:40945 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 06:28:30 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:30 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:30.383921 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:30.376124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:47905)
0619 06:28:30.376632 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:30.376659 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:30.376763 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:30.377054 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:30.377069 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:30.377367 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:30.377523 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:30.377530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.378829 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.378835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:30.379748 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:30.379759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.380385 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.380389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:30.380519 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:30.381091 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:30.381107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:30.383682 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":260,"threads_started":1}
W20260619 06:28:30.384079 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:30.376255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:47905 (local address 127.0.97.126:44509)
0619 06:28:30.376423 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:30.376427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:30.376728 (+ 301us) server_negotiation.cc:407] Connection header received
0619 06:28:30.376854 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:30.376857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:30.376924 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:30.377002 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:30.377655 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.378691 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.379867 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.380274 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.380364 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:30.383858 (+ 3494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:47905: BlockingRecv error: recv got EOF from 127.0.97.67:47905 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 06:28:30 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:30 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:30.528149 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:30.519684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:56135)
0619 06:28:30.520335 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:30.520352 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:30.520554 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:30.520886 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:30.520890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:30.521342 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:30.521484 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:30.521493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.522515 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.522523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:30.523510 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:30.523522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.524233 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.524238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:30.524424 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:30.525096 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:30.525115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:30.527863 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":422,"threads_started":1}
W20260619 06:28:30.528306 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:30.519720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56135 (local address 127.0.97.126:44509)
0619 06:28:30.519979 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:30.519995 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:30.520448 (+ 453us) server_negotiation.cc:407] Connection header received
0619 06:28:30.520672 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:30.520677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:30.520748 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:30.520850 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:30.521627 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.522383 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.523652 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.524072 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.524222 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:30.528084 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:56135: BlockingRecv error: recv got EOF from 127.0.97.65:56135 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 19 06:28:30 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:30 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:30.949331 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:30.941923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:41451)
0619 06:28:30.942348 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:30.942364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:30.942471 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:30.942732 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:30.942735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:30.943073 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:30.943213 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:30.943220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.944165 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.944169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:30.945151 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:30.945163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.945788 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.945792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:30.945920 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:30.946467 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:30.946483 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:30.949106 (+ 2623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":238,"threads_started":1}
W20260619 06:28:30.949415 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:30.941965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:41451 (local address 127.0.97.126:44509)
0619 06:28:30.942183 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:30.942189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:30.942440 (+ 251us) server_negotiation.cc:407] Connection header received
0619 06:28:30.942551 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:30.942554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:30.942610 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:30.942693 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:30.943327 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.944045 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.945262 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:30.945674 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:30.945765 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:30.949246 (+ 3481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:41451: BlockingRecv error: recv got EOF from 127.0.97.66:41451 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 06:28:31 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:31 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:31.392714 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:31.384808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:44501)
0619 06:28:31.385303 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:31.385320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:31.385434 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:31.385718 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:31.385721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:31.386065 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:31.386211 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:31.386218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.387306 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.387313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:31.388346 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:31.388358 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.389019 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.389025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:31.389252 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:31.389810 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:31.389826 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:31.392353 (+ 2527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":300,"threads_started":1}
W20260619 06:28:31.392761 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:31.384976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:44501 (local address 127.0.97.126:44509)
0619 06:28:31.385238 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:31.385245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:31.385394 (+ 149us) server_negotiation.cc:407] Connection header received
0619 06:28:31.385527 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:31.385530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:31.385598 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:31.385676 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:31.386323 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.387170 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.388487 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.388882 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.389007 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:31.392527 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:44501: BlockingRecv error: recv got EOF from 127.0.97.67:44501 (error 108)
Metrics: {"server-negotiator.queue_time_us":138}
Jun 19 06:28:31 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:31 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:31.537185 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:31.529134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:33927)
0619 06:28:31.529659 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:31.529676 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:31.529781 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:31.530105 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:31.530110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:31.530499 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:31.530659 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:31.530668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.531709 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.531716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:31.532614 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:31.532626 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.533347 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.533353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:31.533517 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:31.534047 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:31.534063 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:31.536827 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":251,"threads_started":1}
W20260619 06:28:31.537297 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:31.529179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33927 (local address 127.0.97.126:44509)
0619 06:28:31.529330 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:31.529334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:31.529747 (+ 413us) server_negotiation.cc:407] Connection header received
0619 06:28:31.529897 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:31.529901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:31.529956 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:31.530077 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:31.530796 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.531579 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.532752 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.533202 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.533327 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:31.537043 (+ 3716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:33927: BlockingRecv error: recv got EOF from 127.0.97.65:33927 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:28:31 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:31 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:31.958786 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:31.950454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:56323)
0619 06:28:31.951147 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:31.951161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:31.951335 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:31.951741 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:31.951745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:31.952208 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:31.952411 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:31.952419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.953617 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.953621 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:31.954523 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:31.954534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.955175 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.955179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:31.955290 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:31.955860 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:31.955876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:31.958551 (+ 2675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":279,"threads_started":1}
W20260619 06:28:31.958909 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:31.950655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:56323 (local address 127.0.97.126:44509)
0619 06:28:31.950869 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:31.950875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:31.951253 (+ 378us) server_negotiation.cc:407] Connection header received
0619 06:28:31.951480 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:31.951486 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:31.951575 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:31.951680 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:31.952542 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.953504 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.954645 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:31.955054 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:31.955147 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:31.958719 (+ 3572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:56323: BlockingRecv error: recv got EOF from 127.0.97.66:56323 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 19 06:28:32 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:32 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:32.401758 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:32.393656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:40545)
0619 06:28:32.394168 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:32.394184 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:32.394287 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:32.394580 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:32.394584 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:32.394924 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:32.395071 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:32.395079 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.396254 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.396258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:32.397464 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:32.397482 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.398187 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.398190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:32.398300 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:32.398817 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:32.398833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:32.401511 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":246,"threads_started":1}
W20260619 06:28:32.401952 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:32.393714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:40545 (local address 127.0.97.126:44509)
0619 06:28:32.393885 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:32.393891 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:32.394255 (+ 364us) server_negotiation.cc:407] Connection header received
0619 06:28:32.394377 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:32.394381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:32.394462 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:32.394545 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:32.395225 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.396117 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.397628 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.398064 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.398169 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:32.401684 (+ 3515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:40545: BlockingRecv error: recv got EOF from 127.0.97.67:40545 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 06:28:32 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:32 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:32.545645 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:32.538177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:34217)
0619 06:28:32.538698 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:32.538718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:32.538824 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:32.539057 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:32.539060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:32.539408 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:32.539571 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:32.539579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.540498 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.540501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:32.541428 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:32.541459 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.542127 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.542131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:32.542240 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:32.542775 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:32.542792 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:32.545428 (+ 2636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":302,"threads_started":1}
W20260619 06:28:32.545853 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:32.538204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34217 (local address 127.0.97.126:44509)
0619 06:28:32.538398 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:32.538402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:32.538786 (+ 384us) server_negotiation.cc:407] Connection header received
0619 06:28:32.538911 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:32.538914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:32.538970 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:32.539044 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:32.539682 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.540362 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.541585 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.542004 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.542105 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:32.545631 (+ 3526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:34217: BlockingRecv error: recv got EOF from 127.0.97.65:34217 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:28:32 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:32 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:32.969162 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:32.959724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:59775)
0619 06:28:32.960240 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:32.960256 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:32.960364 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:32.960763 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:32.960768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:32.961273 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:32.961512 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:32.961524 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.962454 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.962459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:32.963957 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:32.963971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.964682 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.964688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:32.964863 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:32.965498 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:32.965519 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:32.968810 (+ 3291us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":275,"threads_started":1}
W20260619 06:28:32.969245 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:32.959799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:59775 (local address 127.0.97.126:44509)
0619 06:28:32.959955 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:32.959961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:32.960331 (+ 370us) server_negotiation.cc:407] Connection header received
0619 06:28:32.960492 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:32.960496 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:32.960605 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:32.960694 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:32.961678 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.962336 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.964104 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:32.964532 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:32.964621 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:32.968998 (+ 4377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:59775: BlockingRecv error: recv got EOF from 127.0.97.66:59775 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 06:28:33 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:33 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:33.413509 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:33.402847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:36963)
0619 06:28:33.403369 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:33.403385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:33.403489 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:33.403756 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:33.403758 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:33.404212 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:33.404352 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:33.404360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.405510 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.405515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:33.406417 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:33.406428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.407143 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.407149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:33.407314 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:33.410426 (+ 3112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:33.410443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:33.413293 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":291,"threads_started":1}
W20260619 06:28:33.413668 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:33.402934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:36963 (local address 127.0.97.126:44509)
0619 06:28:33.403082 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:33.403086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:33.403451 (+ 365us) server_negotiation.cc:407] Connection header received
0619 06:28:33.403576 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:33.403579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:33.403634 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:33.403711 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:33.404513 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.405372 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.406558 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.406983 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.407130 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:33.413451 (+ 6321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:36963: BlockingRecv error: recv got EOF from 127.0.97.67:36963 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 19 06:28:33 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:33 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:33.554167 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:33.546575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:33119)
0619 06:28:33.547090 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:33.547107 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:33.547215 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:33.547487 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:33.547491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:33.547862 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:33.548023 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:33.548031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.548963 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.548966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:33.549945 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:33.549958 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.550573 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.550577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:33.550679 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:33.551262 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:33.551279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:33.553918 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":229,"threads_started":1}
W20260619 06:28:33.554368 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:33.546599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33119 (local address 127.0.97.126:44509)
0619 06:28:33.546817 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:33.546826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:33.547186 (+ 360us) server_negotiation.cc:407] Connection header received
0619 06:28:33.547308 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:33.547312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:33.547370 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:33.547445 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:33.548149 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.548850 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.550062 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.550454 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.550542 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:33.554118 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:33119: BlockingRecv error: recv got EOF from 127.0.97.65:33119 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:28:33 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:33 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:33.978641 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:33.970253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:51405)
0619 06:28:33.970826 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:33.970841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:33.970993 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:33.971356 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:33.971360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:33.971770 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:33.971922 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:33.971930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.972866 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.972872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:33.973806 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:33.973818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.974567 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.974572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:33.974739 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:33.975281 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:33.975298 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:33.978286 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":266,"threads_started":1}
W20260619 06:28:33.978819 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:33.970307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51405 (local address 127.0.97.126:44509)
0619 06:28:33.970550 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:33.970558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:33.970928 (+ 370us) server_negotiation.cc:407] Connection header received
0619 06:28:33.971142 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:33.971148 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:33.971215 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:33.971318 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:33.972067 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.972721 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.973946 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:33.974414 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:33.974559 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:33.978553 (+ 3994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:51405: BlockingRecv error: recv got EOF from 127.0.97.66:51405 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 19 06:28:34 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:34 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:34.422221 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:34.414338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:44201)
0619 06:28:34.414852 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:34.414867 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:34.415020 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:34.415358 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:34.415362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:34.415793 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:34.415953 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:34.415961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.416963 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.416970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:34.417902 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:34.417914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.418607 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.418613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:34.418789 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:34.419353 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:34.419369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:34.422008 (+ 2639us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":229,"threads_started":1}
W20260619 06:28:34.422384 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:34.414422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:44201 (local address 127.0.97.126:44509)
0619 06:28:34.414613 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:34.414621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:34.414953 (+ 332us) server_negotiation.cc:407] Connection header received
0619 06:28:34.415143 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:34.415148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:34.415221 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:34.415322 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:34.416103 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.416826 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.418050 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.418455 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.418594 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:34.422180 (+ 3586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:44201: BlockingRecv error: recv got EOF from 127.0.97.67:44201 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 06:28:34 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:34 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:34.562865 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:34.555043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:36395)
0619 06:28:34.555602 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:34.555619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:34.555731 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:34.556007 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:34.556011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:34.556371 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:34.556533 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:34.556541 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.557569 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.557573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:34.558447 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:34.558457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.559133 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.559138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:34.559246 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:34.559804 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:34.559820 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:34.562555 (+ 2735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":339,"threads_started":1}
W20260619 06:28:34.563060 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:34.555333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:36395 (local address 127.0.97.126:44509)
0619 06:28:34.555581 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:34.555587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:34.555694 (+ 107us) server_negotiation.cc:407] Connection header received
0619 06:28:34.555827 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:34.555830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:34.555888 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:34.555965 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:34.556658 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.557440 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.558582 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.558992 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.559085 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:34.562782 (+ 3697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:36395: BlockingRecv error: recv got EOF from 127.0.97.65:36395 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 19 06:28:34 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:34 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:34.988256 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:34.979709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:60745)
0619 06:28:34.980221 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:34.980237 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:34.980355 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:34.980708 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:34.980711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:34.981193 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:34.981333 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:34.981342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.982595 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.982599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:34.983558 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:34.983569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.984203 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.984207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:34.984316 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:34.984990 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:34.985027 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:34.988042 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":261,"threads_started":1}
W20260619 06:28:34.988526 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:34.979828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60745 (local address 127.0.97.126:44509)
0619 06:28:34.980006 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:34.980012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:34.980309 (+ 297us) server_negotiation.cc:407] Connection header received
0619 06:28:34.980493 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:34.980496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:34.980558 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:34.980655 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:34.981456 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.982367 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.983675 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:34.984066 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:34.984158 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:34.988236 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:60745: BlockingRecv error: recv got EOF from 127.0.97.66:60745 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:28:35 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:35 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:35.431294 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:35.423125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:49953)
0619 06:28:35.423632 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:35.423648 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:35.423752 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:35.424027 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:35.424029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:35.424397 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:35.424531 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:35.424538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.425565 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.425568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:35.426445 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:35.426456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.427317 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.427324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:35.427478 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:35.428082 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:35.428098 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:35.431021 (+ 2923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":263,"threads_started":1}
W20260619 06:28:35.431459 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:35.423222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:49953 (local address 127.0.97.126:44509)
0619 06:28:35.423381 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:35.423385 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:35.423712 (+ 327us) server_negotiation.cc:407] Connection header received
0619 06:28:35.423840 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:35.423844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:35.423898 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:35.423979 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:35.424636 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.425452 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.426599 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.427130 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.427259 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:35.431210 (+ 3951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:49953: BlockingRecv error: recv got EOF from 127.0.97.67:49953 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:28:35 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:35 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:35.572443 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:35.563892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:34417)
0619 06:28:35.564470 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:35.564488 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:35.564610 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:35.564900 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:35.564904 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:35.565399 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:35.565573 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:35.565583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.566509 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.566513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:35.567646 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:35.567659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.568384 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.568389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:35.568520 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:35.569118 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:35.569151 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:35.572216 (+ 3065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":308,"threads_started":1}
W20260619 06:28:35.572537 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:35.563956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34417 (local address 127.0.97.126:44509)
0619 06:28:35.564140 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:35.564144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:35.564565 (+ 421us) server_negotiation.cc:407] Connection header received
0619 06:28:35.564705 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:35.564708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:35.564768 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:35.564845 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:35.565699 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.566373 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.567773 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.568242 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.568337 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:35.572342 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:34417: BlockingRecv error: recv got EOF from 127.0.97.65:34417 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:28:35 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:35 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:35.996718 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:35.989146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:45729)
0619 06:28:35.989660 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:35.989675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:35.989779 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:35.990049 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:35.990052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:35.990428 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:35.990598 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:35.990606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.991528 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.991531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:35.992401 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:35.992412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.993047 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.993051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:35.993245 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:35.993861 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:35.993878 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:35.996460 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":257,"threads_started":1}
W20260619 06:28:35.996800 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:35.989186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:45729 (local address 127.0.97.126:44509)
0619 06:28:35.989336 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:35.989341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:35.989741 (+ 400us) server_negotiation.cc:407] Connection header received
0619 06:28:35.989878 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:35.989881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:35.989935 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:35.990010 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:35.990714 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.991427 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.992514 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:35.992916 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:35.993005 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:35.996627 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:45729: BlockingRecv error: recv got EOF from 127.0.97.66:45729 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:28:36 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:36 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:36.440052 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:36.432229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:35005)
0619 06:28:36.432738 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:36.432754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:36.432862 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:36.433225 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:36.433229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:36.433604 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:36.433759 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:36.433767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.434692 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.434696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:36.435616 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:36.435627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.436241 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.436245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:36.436359 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:36.437004 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:36.437021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:36.439666 (+ 2645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":291,"threads_started":1}
W20260619 06:28:36.440088 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:36.432278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35005 (local address 127.0.97.126:44509)
0619 06:28:36.432490 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:36.432494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:36.432849 (+ 355us) server_negotiation.cc:407] Connection header received
0619 06:28:36.432976 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:36.432980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:36.433038 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:36.433191 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:36.433876 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.434582 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.435730 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.436123 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.436209 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:36.439851 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:35005: BlockingRecv error: recv got EOF from 127.0.97.67:35005 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 19 06:28:36 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:36 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:36.581468 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:36.573452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:39637)
0619 06:28:36.573936 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:36.573952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:36.574129 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:36.574454 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:36.574458 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:36.574843 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:36.574988 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:36.574997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.575987 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.575995 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:36.576909 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:36.576919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.577638 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.577643 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:36.577820 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:36.578419 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:36.578435 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:36.581081 (+ 2646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":253,"threads_started":1}
W20260619 06:28:36.581563 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:36.573515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:39637 (local address 127.0.97.126:44509)
0619 06:28:36.573745 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:36.573753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:36.574064 (+ 311us) server_negotiation.cc:407] Connection header received
0619 06:28:36.574248 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:36.574255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:36.574317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:36.574432 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:36.575138 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.575859 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.577046 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.577487 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.577622 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:36.581308 (+ 3686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:39637: BlockingRecv error: recv got EOF from 127.0.97.65:39637 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 19 06:28:36 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:36 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:37.005723 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:36.997615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:46027)
0619 06:28:36.998083 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:36.998098 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:36.998202 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:36.998474 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:36.998477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:36.998797 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:36.998954 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:36.998974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.999910 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:36.999913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:37.000803 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:37.000812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.001603 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.001608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:37.001726 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:37.002293 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:37.002309 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:37.005490 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":273,"threads_started":1}
W20260619 06:28:37.005966 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:36.997716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46027 (local address 127.0.97.126:44509)
0619 06:28:36.997982 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:36.997988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:36.998173 (+ 185us) server_negotiation.cc:407] Connection header received
0619 06:28:36.998292 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:36.998295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:36.998352 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:36.998432 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:36.999078 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:36.999730 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.000915 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.001463 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.001573 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:37.005671 (+ 4098us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:46027: BlockingRecv error: recv got EOF from 127.0.97.66:46027 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 19 06:28:37 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:37 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:37.449116 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:37.440975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:57277)
0619 06:28:37.441597 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:37.441613 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:37.441726 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:37.442011 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:37.442014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:37.442346 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:37.442504 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:37.442512 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.443479 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.443483 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:37.444400 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:37.444412 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.445031 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.445035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:37.445222 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:37.445880 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:37.445896 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:37.448808 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":400,"threads_started":1}
W20260619 06:28:37.449302 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:37.441247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:57277 (local address 127.0.97.126:44509)
0619 06:28:37.441511 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:37.441515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:37.441686 (+ 171us) server_negotiation.cc:407] Connection header received
0619 06:28:37.441818 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:37.441822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:37.441877 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:37.441957 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:37.442647 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.443366 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.444515 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.444905 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.444992 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:37.448991 (+ 3999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:57277: BlockingRecv error: recv got EOF from 127.0.97.67:57277 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
Jun 19 06:28:37 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:37 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:37.591001 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:37.582423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:42427)
0619 06:28:37.582935 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:37.582952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:37.583062 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:37.583439 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:37.583443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:37.583814 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:37.584006 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:37.584014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.584973 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.584977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:37.585960 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:37.585976 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.586618 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.586622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:37.586731 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:37.587432 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:37.587451 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:37.590582 (+ 3131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":289,"threads_started":1}
W20260619 06:28:37.591073 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:37.582522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42427 (local address 127.0.97.126:44509)
0619 06:28:37.582694 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:37.582698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:37.583025 (+ 327us) server_negotiation.cc:407] Connection header received
0619 06:28:37.583154 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:37.583157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:37.583213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:37.583292 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:37.584136 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.584858 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.586096 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:37.586501 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:37.586588 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:37.590837 (+ 4249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:42427: BlockingRecv error: recv got EOF from 127.0.97.65:42427 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 06:28:38 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:38 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:38.014784 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:38.006594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:44101)
0619 06:28:38.007068 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:38.007083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:38.007241 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:38.007574 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:38.007579 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:38.008007 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:38.008195 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:38.008203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.009268 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.009274 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:38.010174 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:38.010185 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.010924 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.010930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:38.011107 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:38.011787 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:38.011803 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:38.014451 (+ 2648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":243,"threads_started":1}
W20260619 06:28:38.014866 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:38.006682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44101 (local address 127.0.97.126:44509)
0619 06:28:38.006894 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:38.006902 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:38.007174 (+ 272us) server_negotiation.cc:407] Connection header received
0619 06:28:38.007384 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:38.007388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:38.007443 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:38.007544 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:38.008331 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.009142 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.010308 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.010736 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.010873 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:38.014634 (+ 3761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:44101: BlockingRecv error: recv got EOF from 127.0.97.66:44101 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 06:28:38 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:38 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:38.462157 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:38.450046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:43881)
0619 06:28:38.450507 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:38.450523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:38.450635 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:38.450913 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:38.450916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:38.451285 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:38.451522 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:38.451532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.452841 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.452844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:38.453797 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:38.453808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.454560 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.454564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:38.454672 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:38.459008 (+ 4336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:38.459025 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:38.461912 (+ 2887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":270,"threads_started":1}
W20260619 06:28:38.462301 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:38.450113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:43881 (local address 127.0.97.126:44509)
0619 06:28:38.450335 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:38.450342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:38.450600 (+ 258us) server_negotiation.cc:407] Connection header received
0619 06:28:38.450726 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:38.450730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:38.450786 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:38.450870 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:38.451663 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.452721 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.453945 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.454432 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.454524 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:38.462073 (+ 7549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:43881: BlockingRecv error: recv got EOF from 127.0.97.67:43881 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:28:38 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:38 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:38.600234 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:38.592049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:45327)
0619 06:28:38.592552 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:38.592567 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:38.592672 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:38.592965 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:38.592968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:38.593389 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:38.593537 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:38.593546 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.594512 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.594515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:38.595416 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:38.595426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.596084 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.596087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:38.596206 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:38.597002 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:38.597018 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:38.600013 (+ 2995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":299,"threads_started":1}
W20260619 06:28:38.600452 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:38.592109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45327 (local address 127.0.97.126:44509)
0619 06:28:38.592290 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:38.592294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:38.592639 (+ 345us) server_negotiation.cc:407] Connection header received
0619 06:28:38.592775 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:38.592779 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:38.592835 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:38.592915 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:38.593653 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.594389 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.595530 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:38.595967 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:38.596058 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:38.600181 (+ 4123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:45327: BlockingRecv error: recv got EOF from 127.0.97.65:45327 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:28:39 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:39 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:39.023857 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:39.015733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:39291)
0619 06:28:39.016217 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:39.016234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:39.016348 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:39.016622 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:39.016625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:39.016993 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:39.017211 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:39.017220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.018359 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.018365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:39.019630 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:39.019646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.020283 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.020287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:39.020405 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:39.020981 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:39.021004 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:39.023636 (+ 2632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
W20260619 06:28:39.024008 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:39.015838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39291 (local address 127.0.97.126:44509)
0619 06:28:39.016077 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:39.016083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:39.016312 (+ 229us) server_negotiation.cc:407] Connection header received
0619 06:28:39.016442 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:39.016445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:39.016504 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:39.016580 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:39.017335 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.018210 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.019764 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.020169 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.020260 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:39.023792 (+ 3532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:39291: BlockingRecv error: recv got EOF from 127.0.97.66:39291 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Jun 19 06:28:39 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:39 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:39.471449 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:39.462958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:57769)
0619 06:28:39.463437 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:39.463452 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:39.463577 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:39.463966 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:39.463972 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:39.464448 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:39.464637 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:39.464663 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.465825 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.465829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:39.466830 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:39.466840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.467465 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.467469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:39.467683 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:39.468359 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:39.468376 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:39.471094 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":276,"threads_started":1}
W20260619 06:28:39.471570 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:39.463049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:57769 (local address 127.0.97.126:44509)
0619 06:28:39.463204 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:39.463208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:39.463536 (+ 328us) server_negotiation.cc:407] Connection header received
0619 06:28:39.463682 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:39.463686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:39.463782 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:39.463899 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:39.464786 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.465691 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.466951 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.467350 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.467449 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:39.471326 (+ 3877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:57769: BlockingRecv error: recv got EOF from 127.0.97.67:57769 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 19 06:28:39 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:39 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:39.609350 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:39.601157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:47927)
0619 06:28:39.601610 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:39.601626 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:39.601750 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:39.602026 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:39.602029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:39.602400 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:39.602546 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:39.602554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.603482 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.603485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:39.604378 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:39.604388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.605391 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.605400 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:39.605596 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:39.606229 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:39.606245 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:39.609017 (+ 2772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
W20260619 06:28:39.609421 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:39.601215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47927 (local address 127.0.97.126:44509)
0619 06:28:39.601455 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:39.601461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:39.601703 (+ 242us) server_negotiation.cc:407] Connection header received
0619 06:28:39.601846 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:39.601850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:39.601910 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:39.601990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:39.602661 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.603357 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.604564 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:39.605146 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:39.605339 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:39.609220 (+ 3881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:47927: BlockingRecv error: recv got EOF from 127.0.97.65:47927 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 19 06:28:40 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:40 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:40.033576 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:40.024831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:45307)
0619 06:28:40.025382 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:40.025398 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:40.025523 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:40.025844 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:40.025847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:40.026218 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:40.026370 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:40.026378 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.027497 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.027504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:40.028647 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:40.028661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.029572 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.029577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:40.029791 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:40.030497 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:40.030516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:40.033298 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":315,"threads_started":1}
W20260619 06:28:40.033661 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:40.025032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:45307 (local address 127.0.97.126:44509)
0619 06:28:40.025230 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:40.025235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:40.025506 (+ 271us) server_negotiation.cc:407] Connection header received
0619 06:28:40.025632 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:40.025636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:40.025701 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:40.025794 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:40.026505 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.027337 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.028777 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.029417 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.029527 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:40.033473 (+ 3946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:45307: BlockingRecv error: recv got EOF from 127.0.97.66:45307 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 19 06:28:40 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:40 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:40.480499 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:40.472372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:54347)
0619 06:28:40.472851 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:40.472866 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:40.472985 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:40.473337 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:40.473340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:40.473692 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:40.473839 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:40.473846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.475072 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.475079 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:40.476001 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:40.476013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.476725 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.476732 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:40.476902 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:40.477541 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:40.477559 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:40.480166 (+ 2607us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
W20260619 06:28:40.480537 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:40.472477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:54347 (local address 127.0.97.126:44509)
0619 06:28:40.472722 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:40.472728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:40.472958 (+ 230us) server_negotiation.cc:407] Connection header received
0619 06:28:40.473137 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:40.473140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:40.473220 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:40.473302 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:40.473992 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.474938 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.476157 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.476554 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.476690 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:40.480331 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:54347: BlockingRecv error: recv got EOF from 127.0.97.67:54347 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Jun 19 06:28:40 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:40 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:40.618288 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:40.610251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:53405)
0619 06:28:40.610761 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:40.610777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:40.610908 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:40.611218 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:40.611221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:40.611567 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:40.611734 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:40.611742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.612761 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.612765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:40.613701 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:40.613713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.614374 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.614378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:40.614487 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:40.615053 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:40.615069 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:40.618060 (+ 2991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":276,"threads_started":1}
W20260619 06:28:40.618552 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:40.610293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53405 (local address 127.0.97.126:44509)
0619 06:28:40.610511 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:40.610515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:40.610843 (+ 328us) server_negotiation.cc:407] Connection header received
0619 06:28:40.611014 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:40.611017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:40.611094 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:40.611174 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:40.611860 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.612640 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.613816 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:40.614249 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:40.614347 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:40.618256 (+ 3909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:53405: BlockingRecv error: recv got EOF from 127.0.97.65:53405 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 19 06:28:41 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:41 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:41.042537 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:41.034511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:50585)
0619 06:28:41.034984 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:41.035004 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:41.035157 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:41.035490 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:41.035494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:41.035874 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:41.036040 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:41.036048 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.037086 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.037093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:41.038016 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:41.038027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.038779 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.038784 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:41.038980 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:41.039566 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:41.039582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:41.042316 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
W20260619 06:28:41.042717 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:41.034552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50585 (local address 127.0.97.126:44509)
0619 06:28:41.034734 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:41.034741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:41.035108 (+ 367us) server_negotiation.cc:407] Connection header received
0619 06:28:41.035293 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:41.035299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:41.035363 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:41.035459 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:41.036194 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.036913 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.038172 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.038593 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.038765 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:41.042488 (+ 3723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:50585: BlockingRecv error: recv got EOF from 127.0.97.66:50585 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 06:28:41 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:41 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.67@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:41.489239 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:41.481408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.67:55749)
0619 06:28:41.481904 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:41.481922 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:41.482029 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:41.482309 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:41.482312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:41.482670 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:41.482852 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:41.482861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.483837 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.483840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:41.484729 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:41.484739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.485393 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.485397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:41.485515 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:41.486128 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:41.486145 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:41.488960 (+ 2815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":276,"threads_started":1}
W20260619 06:28:41.489480 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:41.481445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55749 (local address 127.0.97.126:44509)
0619 06:28:41.481618 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:41.481622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:41.481991 (+ 369us) server_negotiation.cc:407] Connection header received
0619 06:28:41.482117 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:41.482121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:41.482192 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:41.482277 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:41.482969 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.483711 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.484839 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.485272 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.485358 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:41.489221 (+ 3863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.67:55749: BlockingRecv error: recv got EOF from 127.0.97.67:55749 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 19 06:28:41 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:41 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.65@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:41.627493 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:41.619458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.65:56143)
0619 06:28:41.619968 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:41.619983 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:41.620119 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:41.620546 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:41.620551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:41.621122 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:41.621393 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:41.621408 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.622401 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.622405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:41.623295 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:41.623305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.623932 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.623935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:41.624038 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:41.624572 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:41.624588 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:41.627246 (+ 2658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":291,"threads_started":1}
W20260619 06:28:41.627637 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:41.619558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56143 (local address 127.0.97.126:44509)
0619 06:28:41.619713 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:41.619718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:41.620068 (+ 350us) server_negotiation.cc:407] Connection header received
0619 06:28:41.620300 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:41.620305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:41.620407 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:41.620513 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:41.621549 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.622281 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.623407 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:41.623818 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:41.623910 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:41.627416 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.65:56143: BlockingRecv error: recv got EOF from 127.0.97.65:56143 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 19 06:28:42 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
Jun 19 06:28:42 dist-test-slave-4t1f krb5kdc[8646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.66@KRBTEST.COM for oryx/127.0.97.126@KRBTEST.COM, Server not found in Kerberos database
I20260619 06:28:42.051318 9692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:42.043569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44509 (local address 127.0.97.66:50141)
0619 06:28:42.044258 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:42.044278 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:42.044395 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:42.044663 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:42.044666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:42.045036 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:42.045238 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:42.045246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.046177 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.046180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:42.047071 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:42.047082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.047680 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.047684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:42.047793 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:42.048416 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:42.048433 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:42.051076 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.126:44509: Server oryx/127.0.97.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":424,"threads_started":1}
W20260619 06:28:42.051462 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:42.043618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50141 (local address 127.0.97.126:44509)
0619 06:28:42.043829 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:42.043838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:42.044339 (+ 501us) server_negotiation.cc:407] Connection header received
0619 06:28:42.044475 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:42.044478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:42.044534 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:42.044607 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:42.045351 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.046067 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.047180 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.047559 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.047643 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:42.051250 (+ 3607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.66:50141: BlockingRecv error: recv got EOF from 127.0.97.66:50141 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:28:42.324395 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9233
I20260619 06:28:42.336408 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:42.325486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.0.97.65:42333)
0619 06:28:42.326051 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:42.326059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:42.326084 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:28:42.326138 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:42.326144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:42.326370 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:42.326517 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:42.327934 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.328997 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.330307 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.330870 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.331010 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:42.333459 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:42.333467 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:42.333481 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:42.333512 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:42.335731 (+ 2219us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:28:42.336072 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46722: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":294,"threads_started":1}
W20260619 06:28:42.336494 9694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46722: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:28:42.336454 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9233: Not authorized: Client connection negotiation failed: client connection to 127.0.97.65:42333: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:28:42.336652 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9233
I20260619 06:28:42.352857 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9384
I20260619 06:28:42.364216 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:42.353848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.97.66:45299)
0619 06:28:42.354369 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:42.354376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:42.354405 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:28:42.354461 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:42.354467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:42.354667 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:42.354866 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:42.356110 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.357180 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.358369 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.358913 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.359063 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:42.361375 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:42.361384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:42.361397 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:42.361430 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:42.363568 (+ 2138us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:28:42.363892 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39926: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
W20260619 06:28:42.364203 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9384: Not authorized: Client connection negotiation failed: client connection to 127.0.97.66:45299: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:28:42.364327 9695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39926: Request ticket server kudu/127.0.97.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:28:42.364365 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9384
I20260619 06:28:42.380484 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9516
I20260619 06:28:42.391506 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:42.381425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55108 (local address 127.0.97.67:45457)
0619 06:28:42.382013 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:42.382019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:42.382043 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:28:42.382104 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:42.382110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:42.382322 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:42.382471 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:42.383571 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.384562 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.385752 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:42.386267 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:42.386359 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:42.388532 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:42.388540 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:42.388553 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:42.388582 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:42.391051 (+ 2469us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0619 06:28:42.391286 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55108: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":305,"threads_started":1}
W20260619 06:28:42.391644 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9516: Not authorized: Client connection negotiation failed: client connection to 127.0.97.67:45457: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260619 06:28:42.391728 9696 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55108: Request ticket server kudu/127.0.97.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260619 06:28:42.391803 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9516
I20260619 06:28:42.407827 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9164
I20260619 06:28:42.409125 9226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:42.522881 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9164
2026-06-19T06:28:42Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20856 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T06:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:42Z Disabled control of system clock
I20260619 06:28:42.595376 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36779
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36779
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 06:28:42.900964 9701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:42.901257 9701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:42.901343 9701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:42.908099 9701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:42.908186 9701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:42.908222 9701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:42.908281 9701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:42.908315 9701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:42.916968 9701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36779
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:42.918179 9701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:42.919859 9701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:42.931409 9709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:42.931444 9707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:42.931495 9706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:42.932588 9701 server_base.cc:1034] running on GCE node
I20260619 06:28:42.933349 9701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:42.934600 9701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:42.935812 9701 hybrid_clock.cc:648] HybridClock initialized: now 1781850522935781 us; error 46 us; skew 500 ppm
I20260619 06:28:42.936138 9701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:42.938305 9701 webserver.cc:460] Webserver started at http://127.0.97.126:34017/ using document root <none> and password file <none>
I20260619 06:28:42.938951 9701 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:42.939100 9701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:42.939414 9701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:42.942118 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "825ab834f5634699870ab3aeef467d69"
format_stamp: "Formatted at 2026-06-19 06:28:42 on dist-test-slave-4t1f"
server_key: "61147423f7308db832bd701bc12793d9"
server_key_iv: "a78b000b4213668dc4be0293dff502f6"
server_key_version: "encryptionkey@0"
I20260619 06:28:42.942860 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "825ab834f5634699870ab3aeef467d69"
format_stamp: "Formatted at 2026-06-19 06:28:42 on dist-test-slave-4t1f"
server_key: "61147423f7308db832bd701bc12793d9"
server_key_iv: "a78b000b4213668dc4be0293dff502f6"
server_key_version: "encryptionkey@0"
I20260619 06:28:42.947455 9701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:28:42.950485 9715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:42.951699 9701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:28:42.951869 9701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "825ab834f5634699870ab3aeef467d69"
format_stamp: "Formatted at 2026-06-19 06:28:42 on dist-test-slave-4t1f"
server_key: "61147423f7308db832bd701bc12793d9"
server_key_iv: "a78b000b4213668dc4be0293dff502f6"
server_key_version: "encryptionkey@0"
I20260619 06:28:42.952062 9701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:42.977845 9701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:42.978623 9701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:42.978899 9701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:42.996300 9701 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36779
I20260619 06:28:42.996308 9766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36779 every 8 connection(s)
I20260619 06:28:42.997535 9701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:43.001034 9767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:43.002255 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9701
I20260619 06:28:43.002466 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:28:43.003116 389 external_mini_cluster.cc:1428] Setting key 4b3e5e09dd1aa79218975a31eb0db9f3
I20260619 06:28:43.014776 9767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69: Bootstrap starting.
I20260619 06:28:43.016466 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:43.004895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.0.97.126:36779)
0619 06:28:43.005908 (+ 1013us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:43.005918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:43.005956 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:28:43.007477 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:43.007495 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:43.007935 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:43.008252 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:43.008924 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.010196 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.012065 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.012750 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.012884 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:43.014157 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:43.014165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:43.014178 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:43.014818 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:43.014925 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:43.014959 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:43.015044 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:43.015266 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:28:43.015575 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":398,"threads_started":1}
I20260619 06:28:43.018152 9767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:43.019248 9767 log.cc:824] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:43.021741 9767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69: No bootstrap required, opened a new log
I20260619 06:28:43.028461 9767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825ab834f5634699870ab3aeef467d69" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36779 } }
I20260619 06:28:43.028784 9767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:43.028882 9767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825ab834f5634699870ab3aeef467d69, State: Initialized, Role: FOLLOWER
I20260619 06:28:43.029500 9767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825ab834f5634699870ab3aeef467d69" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36779 } }
I20260619 06:28:43.029713 9767 raft_consensus.cc:388] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:43.029879 9767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:43.030082 9767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:43.032094 9767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825ab834f5634699870ab3aeef467d69" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36779 } }
I20260619 06:28:43.032487 9767 leader_election.cc:302] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 825ab834f5634699870ab3aeef467d69; no voters:
I20260619 06:28:43.032975 9767 leader_election.cc:288] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:28:43.033143 9772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:43.033852 9772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [term 1 LEADER]: Becoming Leader. State: Replica: 825ab834f5634699870ab3aeef467d69, State: Running, Role: LEADER
I20260619 06:28:43.034369 9772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825ab834f5634699870ab3aeef467d69" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36779 } }
I20260619 06:28:43.034878 9767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:43.036906 9774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 825ab834f5634699870ab3aeef467d69. Latest consensus state: current_term: 1 leader_uuid: "825ab834f5634699870ab3aeef467d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825ab834f5634699870ab3aeef467d69" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36779 } } }
I20260619 06:28:43.037303 9774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:43.037344 9773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "825ab834f5634699870ab3aeef467d69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "825ab834f5634699870ab3aeef467d69" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36779 } } }
I20260619 06:28:43.037845 9773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:43.039572 9777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:43.044402 9777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:43.053035 9777 catalog_manager.cc:1347] Generated new cluster ID: e24e2555c68943aa8de3513b0db45226
I20260619 06:28:43.053191 9777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:43.059533 9777 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:28:43.060819 9777 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:43.075785 9777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 825ab834f5634699870ab3aeef467d69: Generated new TSK 0
I20260619 06:28:43.076555 9777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:28:43.085529 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36779
--builtin_ntp_servers=127.0.97.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 06:28:43.401477 9791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:43.401770 9791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:43.401932 9791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:43.409386 9791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:43.409494 9791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:43.409674 9791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:43.418505 9791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:43.419656 9791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:43.421459 9791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:43.434448 9796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:43.434417 9797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:43.434484 9799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:43.435137 9791 server_base.cc:1034] running on GCE node
I20260619 06:28:43.435835 9791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:43.436755 9791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:43.437971 9791 hybrid_clock.cc:648] HybridClock initialized: now 1781850523437935 us; error 61 us; skew 500 ppm
I20260619 06:28:43.438282 9791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:43.440559 9791 webserver.cc:460] Webserver started at http://127.0.97.65:44985/ using document root <none> and password file <none>
I20260619 06:28:43.441321 9791 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:43.441471 9791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:43.441780 9791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:43.444425 9791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "aba24862a46f40c0b823c2d0ea04b44c"
format_stamp: "Formatted at 2026-06-19 06:28:43 on dist-test-slave-4t1f"
server_key: "11391c7b59ed36eeeedfa2e290c32abb"
server_key_iv: "ff4582023a70f73b36d8c15dfab1809a"
server_key_version: "encryptionkey@0"
I20260619 06:28:43.445258 9791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "aba24862a46f40c0b823c2d0ea04b44c"
format_stamp: "Formatted at 2026-06-19 06:28:43 on dist-test-slave-4t1f"
server_key: "11391c7b59ed36eeeedfa2e290c32abb"
server_key_iv: "ff4582023a70f73b36d8c15dfab1809a"
server_key_version: "encryptionkey@0"
I20260619 06:28:43.449985 9791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:28:43.453035 9805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:43.454272 9791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:28:43.454443 9791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "aba24862a46f40c0b823c2d0ea04b44c"
format_stamp: "Formatted at 2026-06-19 06:28:43 on dist-test-slave-4t1f"
server_key: "11391c7b59ed36eeeedfa2e290c32abb"
server_key_iv: "ff4582023a70f73b36d8c15dfab1809a"
server_key_version: "encryptionkey@0"
I20260619 06:28:43.454645 9791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:43.475622 9791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:43.476408 9791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:43.476673 9791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:43.477613 9791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:43.479022 9791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:43.479135 9791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:43.479259 9791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:43.479368 9791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:43.511432 9791 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36547
I20260619 06:28:43.511479 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36547 every 8 connection(s)
I20260619 06:28:43.512717 9791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:43.515544 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9791
I20260619 06:28:43.515798 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:28:43.516458 389 external_mini_cluster.cc:1428] Setting key 3b13365173c71cc4c4f588c8bae90091
I20260619 06:28:43.518951 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36779
--builtin_ntp_servers=127.0.97.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:28:43.525241 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:43.515271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34433 (local address 127.0.97.126:36779)
0619 06:28:43.515490 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:43.515496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:43.517199 (+ 1703us) server_negotiation.cc:407] Connection header received
0619 06:28:43.518871 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:43.518876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:43.518957 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:43.519073 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:43.520120 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.520976 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.522555 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.522940 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.523075 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:43.524506 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:43.524511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:43.524514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:43.524530 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:43.524560 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:43.524571 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:43.524643 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:43.524900 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 06:28:43.524966 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:28:43.526674 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:43.515990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36779 (local address 127.0.97.65:34433)
0619 06:28:43.517028 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:43.517102 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:43.518605 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:43.519307 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:43.519315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:43.519401 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:43.519959 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:43.519975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.521152 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.521160 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:43.522390 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:43.522406 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.523170 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.523177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:43.523404 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:43.524149 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:28:43.524155 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:43.524212 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:28:43.524312 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:43.524669 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:43.524679 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:43.524690 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:28:43.524889 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 06:28:43.525341 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":334,"threads_started":1}
I20260619 06:28:43.528133 9918 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36779
I20260619 06:28:43.528406 9918 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:43.529152 9918 heartbeater.cc:511] Master 127.0.97.126:36779 requested a full tablet report, sending...
I20260619 06:28:43.531038 9732 ts_manager.cc:194] Registered new tserver with Master: aba24862a46f40c0b823c2d0ea04b44c (127.0.97.65:36547)
I20260619 06:28:43.532431 9732 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:34433
W20260619 06:28:43.836414 9921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:43.836721 9921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:43.836879 9921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:43.843988 9921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:43.844113 9921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:43.844307 9921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:43.852865 9921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:43.854262 9921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:43.856081 9921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:43.868680 9929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:43.868842 9926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:43.868741 9927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:43.869350 9921 server_base.cc:1034] running on GCE node
I20260619 06:28:43.870049 9921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:43.871089 9921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:43.872285 9921 hybrid_clock.cc:648] HybridClock initialized: now 1781850523872230 us; error 51 us; skew 500 ppm
I20260619 06:28:43.872548 9921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:43.874612 9921 webserver.cc:460] Webserver started at http://127.0.97.66:44715/ using document root <none> and password file <none>
I20260619 06:28:43.875203 9921 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:43.875285 9921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:43.875555 9921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:43.878197 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "f83e47fa47ed4ac69262163c8b732d45"
format_stamp: "Formatted at 2026-06-19 06:28:43 on dist-test-slave-4t1f"
server_key: "6b254dc01dd0f9b0a93e9be399413be0"
server_key_iv: "8cc852d00f09cd05c9132a484daa9c81"
server_key_version: "encryptionkey@0"
I20260619 06:28:43.878894 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "f83e47fa47ed4ac69262163c8b732d45"
format_stamp: "Formatted at 2026-06-19 06:28:43 on dist-test-slave-4t1f"
server_key: "6b254dc01dd0f9b0a93e9be399413be0"
server_key_iv: "8cc852d00f09cd05c9132a484daa9c81"
server_key_version: "encryptionkey@0"
I20260619 06:28:43.883716 9921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:28:43.886813 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:43.888007 9921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:28:43.888128 9921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "f83e47fa47ed4ac69262163c8b732d45"
format_stamp: "Formatted at 2026-06-19 06:28:43 on dist-test-slave-4t1f"
server_key: "6b254dc01dd0f9b0a93e9be399413be0"
server_key_iv: "8cc852d00f09cd05c9132a484daa9c81"
server_key_version: "encryptionkey@0"
I20260619 06:28:43.888286 9921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:43.911015 9921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:43.911800 9921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:43.912027 9921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:43.912892 9921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:43.914283 9921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:43.914379 9921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:43.914455 9921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:43.914510 9921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:43.946942 9921 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:41927
I20260619 06:28:43.947048 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:41927 every 8 connection(s)
I20260619 06:28:43.948222 9921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:43.950868 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 9921
I20260619 06:28:43.951059 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:28:43.951524 389 external_mini_cluster.cc:1428] Setting key 410f67ea37fad39a8314b1c9b36b11ca
I20260619 06:28:43.954510 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36779
--builtin_ntp_servers=127.0.97.84:44773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 06:28:43.961418 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:43.950676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35477 (local address 127.0.97.126:36779)
0619 06:28:43.950892 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:43.950897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:43.952470 (+ 1573us) server_negotiation.cc:407] Connection header received
0619 06:28:43.954728 (+ 2258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:43.954736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:43.954810 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:43.954933 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:43.956037 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.956764 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.958536 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.959104 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.959234 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:43.960655 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:43.960660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:43.960663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:43.960678 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:43.960717 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:43.960729 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:43.960796 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:43.961128 (+ 332us) server_negotiation.cc:299] Negotiation successful
0619 06:28:43.961198 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:28:43.962416 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:43.951303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36779 (local address 127.0.97.66:35477)
0619 06:28:43.952327 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:43.952373 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:43.954428 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:43.955209 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:43.955220 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:43.955323 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:43.955864 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:43.955881 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.956933 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.956938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:43.958377 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:43.958406 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:43.959318 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:43.959326 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:43.959552 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:43.960304 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:28:43.960309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:43.960361 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:28:43.960464 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:43.960845 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:43.960858 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:43.960869 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:28:43.961097 (+ 228us) client_negotiation.cc:240] Negotiation successful
0619 06:28:43.961442 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":317,"threads_started":1}
I20260619 06:28:43.963735 10048 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36779
I20260619 06:28:43.964025 10048 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:43.964861 10048 heartbeater.cc:511] Master 127.0.97.126:36779 requested a full tablet report, sending...
I20260619 06:28:43.966234 9732 ts_manager.cc:194] Registered new tserver with Master: f83e47fa47ed4ac69262163c8b732d45 (127.0.97.66:41927)
I20260619 06:28:43.967056 9732 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:35477
W20260619 06:28:44.271000 10051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:44.271284 10051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:44.271441 10051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:44.278506 10051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:44.278637 10051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:44.278811 10051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:44.287559 10051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:44773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:36779
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:44.288780 10051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:44.290536 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:44.303570 10059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:44.303639 10056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:44.303619 10057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:44.304631 10051 server_base.cc:1034] running on GCE node
I20260619 06:28:44.305230 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:44.306134 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:44.307353 10051 hybrid_clock.cc:648] HybridClock initialized: now 1781850524307310 us; error 57 us; skew 500 ppm
I20260619 06:28:44.307639 10051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:44.309878 10051 webserver.cc:460] Webserver started at http://127.0.97.67:35715/ using document root <none> and password file <none>
I20260619 06:28:44.310472 10051 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:44.310602 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:44.310892 10051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:44.313606 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "fdbb857b89264c139d6cbc313927f5dc"
format_stamp: "Formatted at 2026-06-19 06:28:44 on dist-test-slave-4t1f"
server_key: "8c10b48304f00479a57419639aa907e7"
server_key_iv: "1b2b4a464897db8cc1b28b00007b0ed3"
server_key_version: "encryptionkey@0"
I20260619 06:28:44.314363 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "fdbb857b89264c139d6cbc313927f5dc"
format_stamp: "Formatted at 2026-06-19 06:28:44 on dist-test-slave-4t1f"
server_key: "8c10b48304f00479a57419639aa907e7"
server_key_iv: "1b2b4a464897db8cc1b28b00007b0ed3"
server_key_version: "encryptionkey@0"
I20260619 06:28:44.319296 10051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:28:44.322515 10065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:44.323663 10051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:28:44.323848 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "fdbb857b89264c139d6cbc313927f5dc"
format_stamp: "Formatted at 2026-06-19 06:28:44 on dist-test-slave-4t1f"
server_key: "8c10b48304f00479a57419639aa907e7"
server_key_iv: "1b2b4a464897db8cc1b28b00007b0ed3"
server_key_version: "encryptionkey@0"
I20260619 06:28:44.324052 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:44.334126 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:44.334885 10051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:44.335135 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:44.336063 10051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:44.337400 10051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:44.337515 10051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:44.337637 10051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:44.337730 10051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:44.369319 10051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:41561
I20260619 06:28:44.369355 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:41561 every 8 connection(s)
I20260619 06:28:44.370569 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:44.376654 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10051
I20260619 06:28:44.376850 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:28:44.377297 389 external_mini_cluster.cc:1428] Setting key a63a9ea92eda2e538f5e3349b0832dcd
I20260619 06:28:44.382332 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:44.372975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34229 (local address 127.0.97.126:36779)
0619 06:28:44.373221 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:44.373227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:44.374526 (+ 1299us) server_negotiation.cc:407] Connection header received
0619 06:28:44.376110 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:44.376114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:44.376176 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:44.376267 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:44.377402 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.378252 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.379771 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.380150 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.380289 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:44.381559 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:44.381564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:44.381567 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:44.381582 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:44.381611 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:44.381621 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:44.381716 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:44.381934 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 06:28:44.382021 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260619 06:28:44.383256 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:44.373516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:36779 (local address 127.0.97.67:34229)
0619 06:28:44.374378 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:44.374416 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:44.375891 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:44.376451 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:44.376459 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:44.376543 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:44.377230 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:44.377249 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.378408 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.378420 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:44.379619 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:44.379630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.380372 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.380379 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:44.380598 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:44.381266 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:28:44.381268 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:44.381302 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:28:44.381374 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:44.381725 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:44.381737 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:44.381747 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 06:28:44.381909 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:28:44.382269 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":287,"threads_started":1}
I20260619 06:28:44.384508 10178 heartbeater.cc:347] Connected to a master server at 127.0.97.126:36779
I20260619 06:28:44.384802 10178 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:44.385586 10178 heartbeater.cc:511] Master 127.0.97.126:36779 requested a full tablet report, sending...
I20260619 06:28:44.386854 9732 ts_manager.cc:194] Registered new tserver with Master: fdbb857b89264c139d6cbc313927f5dc (127.0.97.67:41561)
I20260619 06:28:44.387586 9732 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:34229
I20260619 06:28:44.392192 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:28:44.397843 9770 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:44.396378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.97.126:36779)
0619 06:28:44.396578 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:44.396587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:44.396944 (+ 357us) server_negotiation.cc:407] Connection header received
0619 06:28:44.397058 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:44.397100 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:44.397173 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:44.397257 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:44.397644 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60092: BlockingRecv error: recv got EOF from 127.0.0.1:60092 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:28:44.398401 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9791
I20260619 06:28:44.405354 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:44.399241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38142 (local address 127.0.97.65:36547)
0619 06:28:44.399761 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:44.399768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:44.399807 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:28:44.399858 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:44.399863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:44.400028 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:44.400145 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:44.400446 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.401479 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.402609 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.403111 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.403202 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:44.404084 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:44.404090 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:44.404101 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:44.404617 (+ 516us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:44.404659 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:44.404677 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:44.404761 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:44.404866 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:28:44.405015 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":273,"threads_started":1}
I20260619 06:28:44.406404 9913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:44.486119 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9791
I20260619 06:28:44.501922 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9921
I20260619 06:28:44.509790 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:44.502748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.0.97.66:41927)
0619 06:28:44.503261 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:44.503268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:44.503303 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:28:44.503361 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:44.503367 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:44.503588 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:44.503759 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:44.504188 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.505224 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.506436 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.506999 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.507142 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:44.508218 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:44.508226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:44.508239 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:44.508809 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:44.508852 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:44.508870 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:44.508994 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:44.509334 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 06:28:44.509462 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":281,"threads_started":1}
I20260619 06:28:44.510982 10043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:44.592370 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9921
I20260619 06:28:44.607805 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10051
I20260619 06:28:44.615998 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:44.608752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.0.97.67:41561)
0619 06:28:44.609404 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:44.609411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:44.609439 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:28:44.609515 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:44.609522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:44.609731 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:44.609867 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:44.610328 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.611443 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.612737 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:44.613391 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:44.613501 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:44.614523 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:44.614533 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:44.614559 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:44.615171 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:44.615217 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:44.615238 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:44.615343 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:44.615422 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 06:28:44.615555 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":335,"threads_started":1}
I20260619 06:28:44.617358 10173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:44.698184 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10051
I20260619 06:28:44.713735 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 9701
I20260619 06:28:44.714865 9762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:44.815001 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 9701
2026-06-19T06:28:44Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2286 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T06:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:44Z Disabled control of system clock
I20260619 06:28:44.881196 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41221
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:41221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:28:45.188166 10197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:45.188431 10197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:45.188511 10197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:45.195418 10197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:45.195509 10197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:45.195549 10197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:45.195595 10197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:45.195636 10197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:45.205616 10197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:41221
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:45.206733 10197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:45.208457 10197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:45.219887 10203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:45.219878 10202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:45.220180 10205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:45.220902 10197 server_base.cc:1034] running on GCE node
I20260619 06:28:45.221680 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:45.222962 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:45.224205 10197 hybrid_clock.cc:648] HybridClock initialized: now 1781850525224158 us; error 54 us; skew 500 ppm
I20260619 06:28:45.224526 10197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:45.226629 10197 webserver.cc:460] Webserver started at http://127.0.97.126:41975/ using document root <none> and password file <none>
I20260619 06:28:45.227263 10197 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:45.227389 10197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:45.227687 10197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:45.230417 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "2a2dc5a4ca124916801d936ddc668467"
format_stamp: "Formatted at 2026-06-19 06:28:45 on dist-test-slave-4t1f"
server_key: "1fac40961c74b3c1c9ae0b344ddb59b0"
server_key_iv: "3ef0f54dafccc85671ee5b2b666e0908"
server_key_version: "encryptionkey@0"
I20260619 06:28:45.231160 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "2a2dc5a4ca124916801d936ddc668467"
format_stamp: "Formatted at 2026-06-19 06:28:45 on dist-test-slave-4t1f"
server_key: "1fac40961c74b3c1c9ae0b344ddb59b0"
server_key_iv: "3ef0f54dafccc85671ee5b2b666e0908"
server_key_version: "encryptionkey@0"
I20260619 06:28:45.235802 10197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:28:45.238937 10211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:45.240091 10197 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:28:45.240265 10197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "2a2dc5a4ca124916801d936ddc668467"
format_stamp: "Formatted at 2026-06-19 06:28:45 on dist-test-slave-4t1f"
server_key: "1fac40961c74b3c1c9ae0b344ddb59b0"
server_key_iv: "3ef0f54dafccc85671ee5b2b666e0908"
server_key_version: "encryptionkey@0"
I20260619 06:28:45.240466 10197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:45.245955 10197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:45.246642 10197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:45.246918 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:45.264551 10197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:41221
I20260619 06:28:45.264585 10262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:41221 every 8 connection(s)
I20260619 06:28:45.265908 10197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:45.267956 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10197
I20260619 06:28:45.268153 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:28:45.268791 389 external_mini_cluster.cc:1428] Setting key 35866abc365e99ebe384211e67f1739a
I20260619 06:28:45.269785 10263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:45.277207 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:45.270702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48280 (local address 127.0.97.126:41221)
0619 06:28:45.271612 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:45.271621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:45.271660 (+ 39us) server_negotiation.cc:407] Connection header received
0619 06:28:45.273859 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:45.273878 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:45.274270 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:45.274605 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:45.275651 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:45.275656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:45.275668 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:45.275682 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:45.275743 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:45.275761 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:45.275930 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:45.276048 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:28:45.276328 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":322,"threads_started":1}
I20260619 06:28:45.281749 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467: Bootstrap starting.
I20260619 06:28:45.284535 10263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:45.285584 10263 log.cc:824] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:45.288019 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467: No bootstrap required, opened a new log
I20260619 06:28:45.294811 10263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2dc5a4ca124916801d936ddc668467" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41221 } }
I20260619 06:28:45.295130 10263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:45.295221 10263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a2dc5a4ca124916801d936ddc668467, State: Initialized, Role: FOLLOWER
I20260619 06:28:45.295753 10263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2dc5a4ca124916801d936ddc668467" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41221 } }
I20260619 06:28:45.295970 10263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:45.296099 10263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:45.296283 10263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:45.298378 10263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2dc5a4ca124916801d936ddc668467" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41221 } }
I20260619 06:28:45.298843 10263 leader_election.cc:302] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a2dc5a4ca124916801d936ddc668467; no voters:
I20260619 06:28:45.299384 10263 leader_election.cc:288] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:28:45.299521 10268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:45.300192 10268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [term 1 LEADER]: Becoming Leader. State: Replica: 2a2dc5a4ca124916801d936ddc668467, State: Running, Role: LEADER
I20260619 06:28:45.300730 10268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2dc5a4ca124916801d936ddc668467" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41221 } }
I20260619 06:28:45.301324 10263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:45.303546 10270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a2dc5a4ca124916801d936ddc668467. Latest consensus state: current_term: 1 leader_uuid: "2a2dc5a4ca124916801d936ddc668467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2dc5a4ca124916801d936ddc668467" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41221 } } }
I20260619 06:28:45.303987 10270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:45.304375 10269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a2dc5a4ca124916801d936ddc668467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a2dc5a4ca124916801d936ddc668467" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41221 } } }
I20260619 06:28:45.304831 10269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:45.308183 10277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:45.312813 10277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:45.320784 10277 catalog_manager.cc:1347] Generated new cluster ID: d531936717cf4af0bba6704a1af5a94e
I20260619 06:28:45.320914 10277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:45.360373 10277 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:28:45.360910 10277 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:45.369304 10277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a2dc5a4ca124916801d936ddc668467: Generated new TSK 0
I20260619 06:28:45.370265 10277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:28:45.380896 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41221
--builtin_ntp_servers=127.0.97.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 06:28:45.692727 10287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:45.692966 10287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:45.693049 10287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:45.699891 10287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:45.700024 10287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:45.700217 10287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:45.709122 10287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:45.710270 10287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:45.711961 10287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:45.724498 10293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:45.724534 10292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:45.724448 10295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:45.725277 10287 server_base.cc:1034] running on GCE node
I20260619 06:28:45.725827 10287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:45.726692 10287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:45.727921 10287 hybrid_clock.cc:648] HybridClock initialized: now 1781850525727878 us; error 56 us; skew 500 ppm
I20260619 06:28:45.728240 10287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:45.730331 10287 webserver.cc:460] Webserver started at http://127.0.97.65:41033/ using document root <none> and password file <none>
I20260619 06:28:45.730989 10287 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:45.731124 10287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:45.731426 10287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:45.734135 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "6dfb5fa5d9d74242bc95877fc06c9924"
format_stamp: "Formatted at 2026-06-19 06:28:45 on dist-test-slave-4t1f"
server_key: "45861d889314a2a02ff8c933ab7215fc"
server_key_iv: "62e09954e0ee05d81d4253eb6e425f0c"
server_key_version: "encryptionkey@0"
I20260619 06:28:45.734866 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "6dfb5fa5d9d74242bc95877fc06c9924"
format_stamp: "Formatted at 2026-06-19 06:28:45 on dist-test-slave-4t1f"
server_key: "45861d889314a2a02ff8c933ab7215fc"
server_key_iv: "62e09954e0ee05d81d4253eb6e425f0c"
server_key_version: "encryptionkey@0"
I20260619 06:28:45.739775 10287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 06:28:45.743093 10301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:45.744356 10287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:28:45.744510 10287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "6dfb5fa5d9d74242bc95877fc06c9924"
format_stamp: "Formatted at 2026-06-19 06:28:45 on dist-test-slave-4t1f"
server_key: "45861d889314a2a02ff8c933ab7215fc"
server_key_iv: "62e09954e0ee05d81d4253eb6e425f0c"
server_key_version: "encryptionkey@0"
I20260619 06:28:45.744771 10287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:45.750209 10287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:45.750988 10287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:45.751267 10287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:45.752104 10287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:45.753461 10287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:45.753588 10287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:45.753727 10287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:45.753834 10287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:45.785647 10287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:46607
I20260619 06:28:45.785703 10413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:46607 every 8 connection(s)
I20260619 06:28:45.786954 10287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:45.790776 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10287
I20260619 06:28:45.790964 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:28:45.791769 389 external_mini_cluster.cc:1428] Setting key 6fac37a2b93e888a05d2e31981583fd6
I20260619 06:28:45.794879 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41221
--builtin_ntp_servers=127.0.97.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:28:45.795758 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:45.789401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57905 (local address 127.0.97.126:41221)
0619 06:28:45.789835 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:45.789839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:45.791116 (+ 1277us) server_negotiation.cc:407] Connection header received
0619 06:28:45.793317 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:45.793322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:45.793374 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:45.793486 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:45.794913 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:45.794916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:45.794920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:45.794934 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:45.795015 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:45.795034 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:45.795164 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:45.795422 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 06:28:45.795484 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":235,"threads_started":1}
I20260619 06:28:45.796648 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:45.789838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41221 (local address 127.0.97.65:57905)
0619 06:28:45.790903 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:45.790957 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:45.793012 (+ 2055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:45.793728 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:45.793736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:45.793878 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:45.794564 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:28:45.794569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:45.794622 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:28:45.794738 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:45.795226 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:45.795234 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:45.795240 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:28:45.795504 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 06:28:45.795757 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":254,"threads_started":1}
I20260619 06:28:45.798166 10414 heartbeater.cc:347] Connected to a master server at 127.0.97.126:41221
I20260619 06:28:45.798473 10414 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:45.799109 10414 heartbeater.cc:511] Master 127.0.97.126:41221 requested a full tablet report, sending...
I20260619 06:28:45.801342 10228 ts_manager.cc:194] Registered new tserver with Master: 6dfb5fa5d9d74242bc95877fc06c9924 (127.0.97.65:46607)
W20260619 06:28:46.115269 10418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:46.115753 10418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:46.115935 10418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:46.124533 10418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:46.124622 10418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:46.124770 10418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:46.133913 10418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:46.135123 10418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:46.136837 10418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:46.149464 10423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:46.149700 10424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:46.149677 10426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:46.150461 10418 server_base.cc:1034] running on GCE node
I20260619 06:28:46.151023 10418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:46.151857 10418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:46.153115 10418 hybrid_clock.cc:648] HybridClock initialized: now 1781850526153034 us; error 33 us; skew 500 ppm
I20260619 06:28:46.153434 10418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:46.155449 10418 webserver.cc:460] Webserver started at http://127.0.97.66:33867/ using document root <none> and password file <none>
I20260619 06:28:46.156102 10418 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:46.156230 10418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:46.156525 10418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:46.160952 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "f0b1d55357754ab6a01bcf58489c5ebf"
format_stamp: "Formatted at 2026-06-19 06:28:46 on dist-test-slave-4t1f"
server_key: "299f8cf050b025f2a49de066f444a61f"
server_key_iv: "058e350de83a8cac1980513c863648f2"
server_key_version: "encryptionkey@0"
I20260619 06:28:46.161785 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "f0b1d55357754ab6a01bcf58489c5ebf"
format_stamp: "Formatted at 2026-06-19 06:28:46 on dist-test-slave-4t1f"
server_key: "299f8cf050b025f2a49de066f444a61f"
server_key_iv: "058e350de83a8cac1980513c863648f2"
server_key_version: "encryptionkey@0"
I20260619 06:28:46.166574 10418 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:28:46.169780 10432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:46.170950 10418 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:28:46.171154 10418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "f0b1d55357754ab6a01bcf58489c5ebf"
format_stamp: "Formatted at 2026-06-19 06:28:46 on dist-test-slave-4t1f"
server_key: "299f8cf050b025f2a49de066f444a61f"
server_key_iv: "058e350de83a8cac1980513c863648f2"
server_key_version: "encryptionkey@0"
I20260619 06:28:46.171402 10418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:46.176544 10418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:46.177335 10418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:46.177600 10418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:46.178426 10418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:46.179631 10418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:46.179764 10418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:46.179889 10418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:46.179975 10418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:46.211463 10418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:33123
I20260619 06:28:46.211513 10544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:33123 every 8 connection(s)
I20260619 06:28:46.212678 10418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:46.217249 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10418
I20260619 06:28:46.217422 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:28:46.217821 389 external_mini_cluster.cc:1428] Setting key 03b5a6da7a9a0fd88eb7ca4cde6e8c35
I20260619 06:28:46.220659 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41221
--builtin_ntp_servers=127.0.97.84:36677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 06:28:46.220882 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:46.215120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:43255 (local address 127.0.97.126:41221)
0619 06:28:46.215283 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:46.215287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:46.216803 (+ 1516us) server_negotiation.cc:407] Connection header received
0619 06:28:46.218597 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:46.218603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:46.218694 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:46.218808 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:46.220186 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:46.220190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:46.220193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:46.220207 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:46.220237 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:46.220250 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:46.220339 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:46.220565 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 06:28:46.220608 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:28:46.222105 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:46.215613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41221 (local address 127.0.97.66:43255)
0619 06:28:46.216642 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:46.216686 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:46.218329 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:46.219034 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:46.219044 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:46.219129 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:46.219864 (+ 735us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:28:46.219868 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:46.219913 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:28:46.220019 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:46.220351 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:46.220360 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:46.220371 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:28:46.220641 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 06:28:46.220907 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":346,"threads_started":1}
I20260619 06:28:46.223454 10545 heartbeater.cc:347] Connected to a master server at 127.0.97.126:41221
I20260619 06:28:46.223742 10545 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:46.224318 10545 heartbeater.cc:511] Master 127.0.97.126:41221 requested a full tablet report, sending...
I20260619 06:28:46.225703 10228 ts_manager.cc:194] Registered new tserver with Master: f0b1d55357754ab6a01bcf58489c5ebf (127.0.97.66:33123)
W20260619 06:28:46.558313 10548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:46.558593 10548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:46.558686 10548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:46.565719 10548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:46.565817 10548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:46.565930 10548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:46.575042 10548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:36677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:41221
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:46.576356 10548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:46.578188 10548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:46.590921 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:46.590967 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:46.590994 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:46.591676 10548 server_base.cc:1034] running on GCE node
I20260619 06:28:46.592290 10548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:46.593205 10548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:46.594417 10548 hybrid_clock.cc:648] HybridClock initialized: now 1781850526594372 us; error 55 us; skew 500 ppm
I20260619 06:28:46.594715 10548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:28:46.596747 10548 webserver.cc:460] Webserver started at http://127.0.97.67:39039/ using document root <none> and password file <none>
I20260619 06:28:46.597450 10548 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:46.597579 10548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:46.597884 10548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:46.600528 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "7a69609c7ba649318273049b712addd4"
format_stamp: "Formatted at 2026-06-19 06:28:46 on dist-test-slave-4t1f"
server_key: "e12ebb6fbc10564daac27c132a048854"
server_key_iv: "59a501b816325cf7fbd02e566e90a6c6"
server_key_version: "encryptionkey@0"
I20260619 06:28:46.601389 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "7a69609c7ba649318273049b712addd4"
format_stamp: "Formatted at 2026-06-19 06:28:46 on dist-test-slave-4t1f"
server_key: "e12ebb6fbc10564daac27c132a048854"
server_key_iv: "59a501b816325cf7fbd02e566e90a6c6"
server_key_version: "encryptionkey@0"
I20260619 06:28:46.606338 10548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 06:28:46.609447 10562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:46.610641 10548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260619 06:28:46.610818 10548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "7a69609c7ba649318273049b712addd4"
format_stamp: "Formatted at 2026-06-19 06:28:46 on dist-test-slave-4t1f"
server_key: "e12ebb6fbc10564daac27c132a048854"
server_key_iv: "59a501b816325cf7fbd02e566e90a6c6"
server_key_version: "encryptionkey@0"
I20260619 06:28:46.611025 10548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:46.616427 10548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:46.617245 10548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:46.617515 10548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:46.618382 10548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:46.619729 10548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:46.619846 10548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:46.619979 10548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:46.620065 10548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:46.652251 10548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:40763
I20260619 06:28:46.652302 10674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:40763 every 8 connection(s)
I20260619 06:28:46.653541 10548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:46.661774 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:46.656099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:53799 (local address 127.0.97.126:41221)
0619 06:28:46.656268 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:46.656272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:46.657754 (+ 1482us) server_negotiation.cc:407] Connection header received
0619 06:28:46.659767 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:46.659773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:46.659846 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:46.659952 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:46.661182 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:46.661186 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:46.661190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:46.661205 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:46.661241 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:46.661254 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:46.661361 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:46.661513 (+ 152us) server_negotiation.cc:299] Negotiation successful
0619 06:28:46.661560 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:28:46.662668 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:46.656618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41221 (local address 127.0.97.67:53799)
0619 06:28:46.657618 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:46.657654 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:46.659477 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:46.660182 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:46.660189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:46.660275 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:46.660865 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:28:46.660867 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:46.660902 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:28:46.660979 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:46.661342 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:46.661349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:46.661354 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:28:46.661555 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 06:28:46.661738 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":257,"threads_started":1}
I20260619 06:28:46.663563 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10548
I20260619 06:28:46.663726 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:28:46.664009 10675 heartbeater.cc:347] Connected to a master server at 127.0.97.126:41221
I20260619 06:28:46.664290 10675 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:46.664281 389 external_mini_cluster.cc:1428] Setting key cb049145963a7c6780e85639002ea27e
I20260619 06:28:46.664891 10675 heartbeater.cc:511] Master 127.0.97.126:41221 requested a full tablet report, sending...
I20260619 06:28:46.666424 10228 ts_manager.cc:194] Registered new tserver with Master: 7a69609c7ba649318273049b712addd4 (127.0.97.67:40763)
I20260619 06:28:46.667330 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 06:28:46.673264 10416 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:28:46.671691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.0.97.126:41221)
0619 06:28:46.671834 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:46.671838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:46.672332 (+ 494us) server_negotiation.cc:407] Connection header received
0619 06:28:46.672512 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:46.672518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:46.672575 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:46.672675 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:46.672979 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48290: BlockingRecv error: recv got EOF from 127.0.0.1:48290 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:28:46.674093 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10287
I20260619 06:28:46.677522 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:46.675123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52916 (local address 127.0.97.65:46607)
0619 06:28:46.675630 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:46.675636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:46.675660 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:28:46.675712 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:46.675717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:46.675904 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:46.676019 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:46.676830 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:46.676836 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:46.676846 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:46.676858 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:46.676898 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:46.676911 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:46.677007 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:46.677143 (+ 136us) server_negotiation.cc:299] Negotiation successful
0619 06:28:46.677247 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":259,"threads_started":1}
I20260619 06:28:46.678610 10409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:46.759286 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10287
I20260619 06:28:46.775959 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10418
I20260619 06:28:46.779561 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:46.776919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41814 (local address 127.0.97.66:33123)
0619 06:28:46.777458 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:46.777472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:46.777498 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:28:46.777558 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:46.777565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:46.777786 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:46.777918 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:46.778870 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:46.778877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:46.778889 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:46.778906 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:46.778944 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:46.778957 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:46.779058 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:46.779163 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:28:46.779274 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
I20260619 06:28:46.780691 10540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:46.862757 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10418
I20260619 06:28:46.878221 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10548
I20260619 06:28:46.882021 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:46.879149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.0.97.67:40763)
0619 06:28:46.879655 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:46.879662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:46.879691 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:28:46.879751 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:46.879758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:46.879964 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:46.880141 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:46.881303 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:46.881310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:46.881321 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:28:46.881333 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:46.881373 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:28:46.881390 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:46.881565 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:46.881630 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 06:28:46.881732 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":227,"threads_started":1}
I20260619 06:28:46.883196 10670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:46.963982 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10548
I20260619 06:28:46.979432 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10197
I20260619 06:28:46.980562 10258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:47.078945 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10197
2026-06-19T06:28:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2264 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:28:47 dist-test-slave-4t1f krb5kdc[10693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:28:47 dist-test-slave-4t1f krb5kdc[10693](info): set up 2 sockets
Jun 19 06:28:47 dist-test-slave-4t1f krb5kdc[10693](info): commencing operation
krb5kdc: starting...
W20260619 06:28:48.177872 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:28:48 dist-test-slave-4t1f krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:28:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:28:48.386337 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46777
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:48.700506 10706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:48.700780 10706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:48.700871 10706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:48.708071 10706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:28:48.708204 10706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:48.708254 10706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:48.708309 10706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:28:48.708372 10706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:28:48.717746 10706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:48.718871 10706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:48.720106 10706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:48.733335 10711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:48.733299 10712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:48.735165 10714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:48.738768 10706 server_base.cc:1034] running on GCE node
I20260619 06:28:48.739727 10706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:48.741612 10706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:48.745148 10706 hybrid_clock.cc:648] HybridClock initialized: now 1781850528745053 us; error 73 us; skew 500 ppm
I20260619 06:28:48.745502 10706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:48 dist-test-slave-4t1f krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:48.749240 10706 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:28:48.751051 10706 webserver.cc:460] Webserver started at http://127.0.97.126:44229/ using document root <none> and password file <none>
I20260619 06:28:48.751958 10706 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:48.752099 10706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:48.752437 10706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:48.756704 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853"
format_stamp: "Formatted at 2026-06-19 06:28:48 on dist-test-slave-4t1f"
server_key: "25a25905ceecfb705b894ada864888bd"
server_key_iv: "85f3cab2ce0ea69f108f44cbc5373332"
server_key_version: "encryptionkey@0"
I20260619 06:28:48.757813 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853"
format_stamp: "Formatted at 2026-06-19 06:28:48 on dist-test-slave-4t1f"
server_key: "25a25905ceecfb705b894ada864888bd"
server_key_iv: "85f3cab2ce0ea69f108f44cbc5373332"
server_key_version: "encryptionkey@0"
I20260619 06:28:48.765424 10706 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260619 06:28:48.769204 10721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:48.770969 10706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:28:48.771147 10706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853"
format_stamp: "Formatted at 2026-06-19 06:28:48 on dist-test-slave-4t1f"
server_key: "25a25905ceecfb705b894ada864888bd"
server_key_iv: "85f3cab2ce0ea69f108f44cbc5373332"
server_key_version: "encryptionkey@0"
I20260619 06:28:48.771363 10706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:48.794325 10706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:48.798074 10706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:48.798346 10706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:48.816393 10706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:46777
I20260619 06:28:48.816429 10772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:46777 every 8 connection(s)
I20260619 06:28:48.818212 10706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:28:48.821906 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:48.825305 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10706
I20260619 06:28:48.825505 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:28:48.826126 389 external_mini_cluster.cc:1428] Setting key 0f88732fe4c6d15a71a360f0ac62a297
I20260619 06:28:48.834312 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853: Bootstrap starting.
I20260619 06:28:48.837906 10773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:48.839192 10773 log.cc:824] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:28:48 dist-test-slave-4t1f krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:48.842486 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853: No bootstrap required, opened a new log
I20260619 06:28:48.848945 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:48.828017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.97.126:46777)
0619 06:28:48.828821 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:48.828831 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:48.828866 (+ 35us) server_negotiation.cc:407] Connection header received
0619 06:28:48.830329 (+ 1463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:48.830347 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:48.830720 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:48.831022 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:48.832704 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:48.834022 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:48.835705 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:48.836344 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:48.836465 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:48.840914 (+ 4449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:48.840926 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:48.840943 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:48.840989 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:48.844813 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:48.845482 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:48.845491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:48.845499 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:48.845698 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:48.846014 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:48.846019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:48.846023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:48.846674 (+ 651us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:48.846967 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:48.847242 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:28:48.847629 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":383,"threads_started":1}
I20260619 06:28:48.852047 10773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46777 } }
I20260619 06:28:48.852340 10773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:48.852414 10773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75eb2ddc0f6b4b5a8b90b89b93d45853, State: Initialized, Role: FOLLOWER
I20260619 06:28:48.852851 10773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46777 } }
I20260619 06:28:48.853032 10773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:28:48.853212 10773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:28:48.853338 10773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:48.855585 10773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46777 } }
I20260619 06:28:48.855973 10773 leader_election.cc:302] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75eb2ddc0f6b4b5a8b90b89b93d45853; no voters:
I20260619 06:28:48.856460 10773 leader_election.cc:288] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:28:48.856603 10778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:48.857332 10778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [term 1 LEADER]: Becoming Leader. State: Replica: 75eb2ddc0f6b4b5a8b90b89b93d45853, State: Running, Role: LEADER
I20260619 06:28:48.857848 10778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46777 } }
I20260619 06:28:48.858655 10773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:28:48.860802 10779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46777 } } }
I20260619 06:28:48.861182 10779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:48.860782 10780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75eb2ddc0f6b4b5a8b90b89b93d45853. Latest consensus state: current_term: 1 leader_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75eb2ddc0f6b4b5a8b90b89b93d45853" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46777 } } }
I20260619 06:28:48.861474 10780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853 [sys.catalog]: This master's current role is: LEADER
I20260619 06:28:48.866394 10783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:28:48.871871 10783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:28:48.879590 10783 catalog_manager.cc:1347] Generated new cluster ID: fb85be47a1a24272900907a69ae6b036
I20260619 06:28:48.879691 10783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:28:48.900192 10783 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:28:48.901516 10783 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:28:48.912200 10783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853: Generated new TSK 0
I20260619 06:28:48.913156 10783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:28:48.968784 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46777
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:49.288296 10799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:49.288555 10799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:49.288654 10799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:49.296033 10799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:49.296159 10799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:49.296345 10799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:28:49.305703 10799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:49.306886 10799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:49.308209 10799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:49.321725 10804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:49.321790 10805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:49.321954 10807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:49.322635 10799 server_base.cc:1034] running on GCE node
I20260619 06:28:49.323154 10799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:49.323973 10799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:49.325157 10799 hybrid_clock.cc:648] HybridClock initialized: now 1781850529325103 us; error 38 us; skew 500 ppm
I20260619 06:28:49.325405 10799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:49 dist-test-slave-4t1f krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:49.328382 10799 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:28:49.329839 10799 webserver.cc:460] Webserver started at http://127.0.97.65:33243/ using document root <none> and password file <none>
I20260619 06:28:49.330477 10799 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:49.330595 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:49.330901 10799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:49.333659 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "145f9cc5601d41b98393b0dea489af98"
format_stamp: "Formatted at 2026-06-19 06:28:49 on dist-test-slave-4t1f"
server_key: "5892dca32aa1bf96a3f4e0e7183af787"
server_key_iv: "c65da4f55c66c29bb9e5c586ce16e675"
server_key_version: "encryptionkey@0"
I20260619 06:28:49.334401 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "145f9cc5601d41b98393b0dea489af98"
format_stamp: "Formatted at 2026-06-19 06:28:49 on dist-test-slave-4t1f"
server_key: "5892dca32aa1bf96a3f4e0e7183af787"
server_key_iv: "c65da4f55c66c29bb9e5c586ce16e675"
server_key_version: "encryptionkey@0"
I20260619 06:28:49.339264 10799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:28:49.342541 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:49.343719 10799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:28:49.343880 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "145f9cc5601d41b98393b0dea489af98"
format_stamp: "Formatted at 2026-06-19 06:28:49 on dist-test-slave-4t1f"
server_key: "5892dca32aa1bf96a3f4e0e7183af787"
server_key_iv: "c65da4f55c66c29bb9e5c586ce16e675"
server_key_version: "encryptionkey@0"
I20260619 06:28:49.344051 10799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:49.362876 10799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:49.366429 10799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:49.366694 10799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:49.367627 10799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:49.368952 10799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:49.369024 10799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:49.369220 10799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:49.369287 10799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:49.401636 10799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:40845
I20260619 06:28:49.401726 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:40845 every 8 connection(s)
I20260619 06:28:49.402892 10799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:28:49.411208 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10799
I20260619 06:28:49.411394 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:28:49.412187 389 external_mini_cluster.cc:1428] Setting key 72b8f689008b95bc89decacd3210ddad
Jun 19 06:28:49 dist-test-slave-4t1f krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:49.423499 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:49.405335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:33637 (local address 127.0.97.126:46777)
0619 06:28:49.405735 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:49.405739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:49.406748 (+ 1009us) server_negotiation.cc:407] Connection header received
0619 06:28:49.408377 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:49.408381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:49.408442 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:49.408538 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:49.409956 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.410644 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.412936 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.413528 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.413643 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:49.417587 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:49.417592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:49.417595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:49.417622 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:49.420106 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:49.420834 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:49.420841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:49.420850 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:49.420956 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:49.421499 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:49.421502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:49.421504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:49.421743 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:49.421853 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:49.422750 (+ 897us) server_negotiation.cc:299] Negotiation successful
0619 06:28:49.423002 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":204,"threads_started":1}
I20260619 06:28:49.424000 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:49.405907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46777 (local address 127.0.97.65:33637)
0619 06:28:49.406574 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:49.406635 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:49.408118 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:49.408746 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:49.408754 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:49.409285 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:49.409807 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:49.409822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.410826 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.410831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:49.412806 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:49.412816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.413744 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.413749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:49.413995 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:49.414715 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:49.414754 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:49.417389 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:49.420251 (+ 2862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:49.420261 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:49.420275 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:49.420699 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:49.421088 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:49.421096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:49.421100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:49.421350 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:49.421869 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:49.421879 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:49.422337 (+ 458us) client_negotiation.cc:769] Sending connection context
0619 06:28:49.422611 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 06:28:49.422948 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":275,"threads_started":1}
I20260619 06:28:49.425449 10927 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46777
I20260619 06:28:49.425724 10927 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:49.426373 10927 heartbeater.cc:511] Master 127.0.97.126:46777 requested a full tablet report, sending...
I20260619 06:28:49.428243 10738 ts_manager.cc:194] Registered new tserver with Master: 145f9cc5601d41b98393b0dea489af98 (127.0.97.65:40845)
I20260619 06:28:49.429836 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:33637
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:28:49.458350 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46777
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:49.769264 10933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:49.769522 10933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:49.769621 10933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:49.776939 10933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:49.777050 10933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:49.777225 10933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:28:49.786825 10933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:49.787956 10933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:49.789239 10933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:49.801774 10939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:49.801872 10941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:49.801813 10938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:49.802459 10933 server_base.cc:1034] running on GCE node
I20260619 06:28:49.803052 10933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:49.803927 10933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:49.805162 10933 hybrid_clock.cc:648] HybridClock initialized: now 1781850529805107 us; error 52 us; skew 500 ppm
I20260619 06:28:49.805451 10933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:49 dist-test-slave-4t1f krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:49.808305 10933 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:28:49.809698 10933 webserver.cc:460] Webserver started at http://127.0.97.66:33771/ using document root <none> and password file <none>
I20260619 06:28:49.810339 10933 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:49.810465 10933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:49.810753 10933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:49.813516 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "683a8a5dbfa543d391e447a7a5b7ff6b"
format_stamp: "Formatted at 2026-06-19 06:28:49 on dist-test-slave-4t1f"
server_key: "37c32987e059f7814fe2dce23bd9c9fc"
server_key_iv: "8eaf46ccdc3fb2dd59e66a98e4cffb4d"
server_key_version: "encryptionkey@0"
I20260619 06:28:49.814246 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "683a8a5dbfa543d391e447a7a5b7ff6b"
format_stamp: "Formatted at 2026-06-19 06:28:49 on dist-test-slave-4t1f"
server_key: "37c32987e059f7814fe2dce23bd9c9fc"
server_key_iv: "8eaf46ccdc3fb2dd59e66a98e4cffb4d"
server_key_version: "encryptionkey@0"
I20260619 06:28:49.819168 10933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:28:49.822606 10948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:49.824261 10933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:28:49.824395 10933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "683a8a5dbfa543d391e447a7a5b7ff6b"
format_stamp: "Formatted at 2026-06-19 06:28:49 on dist-test-slave-4t1f"
server_key: "37c32987e059f7814fe2dce23bd9c9fc"
server_key_iv: "8eaf46ccdc3fb2dd59e66a98e4cffb4d"
server_key_version: "encryptionkey@0"
I20260619 06:28:49.824587 10933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:49.855506 10933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:49.859187 10933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:49.859409 10933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:49.860236 10933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:49.861490 10933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:49.861558 10933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:49.861653 10933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:49.861696 10933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:49.897397 10933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44857
I20260619 06:28:49.897348 11060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44857 every 8 connection(s)
I20260619 06:28:49.898682 10933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:28:49.907371 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 10933
I20260619 06:28:49.907541 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:28:49.908061 389 external_mini_cluster.cc:1428] Setting key 1de903adca73ddab65c8f6c811f3e3d6
Jun 19 06:28:49 dist-test-slave-4t1f krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:49.918569 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:49.901270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33485 (local address 127.0.97.126:46777)
0619 06:28:49.901437 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:49.901442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:49.902549 (+ 1107us) server_negotiation.cc:407] Connection header received
0619 06:28:49.904212 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:49.904217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:49.904292 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:49.904380 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:49.905877 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.906572 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.908686 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.909259 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.909380 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:49.913145 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:49.913150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:49.913153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:49.913182 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:49.915603 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:49.916437 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:49.916444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:49.916451 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:49.916569 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:49.916969 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:49.916976 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:49.916981 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:49.917279 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:49.917425 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:49.918022 (+ 597us) server_negotiation.cc:299] Negotiation successful
0619 06:28:49.918254 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":58}
I20260619 06:28:49.919450 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:49.901716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46777 (local address 127.0.97.66:33485)
0619 06:28:49.902387 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:49.902429 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:49.903976 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:49.904604 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:49.904612 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:49.905150 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:49.905728 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:49.905745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.906703 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.906707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:49.908542 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:49.908556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:49.909451 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:49.909457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:49.909635 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:49.910370 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:49.910398 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:49.912914 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:49.915733 (+ 2819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:49.915742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:49.915757 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:49.916316 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:49.916684 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:49.916690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:49.916694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:49.916862 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:49.917398 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:49.917406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:49.917714 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 06:28:49.917952 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 06:28:49.918397 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":267,"threads_started":1}
I20260619 06:28:49.920838 11061 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46777
I20260619 06:28:49.921162 11061 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:49.921962 11061 heartbeater.cc:511] Master 127.0.97.126:46777 requested a full tablet report, sending...
I20260619 06:28:49.923539 10738 ts_manager.cc:194] Registered new tserver with Master: 683a8a5dbfa543d391e447a7a5b7ff6b (127.0.97.66:44857)
I20260619 06:28:49.924814 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:33485
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:28:49.956415 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46777
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:41823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:28:50.280881 11066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:28:50.281179 11066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:28:50.281361 11066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:28:50.288938 11066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:28:50.289064 11066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:28:50.289285 11066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:28:50.298638 11066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:41823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:28:50.299819 11066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:28:50.301123 11066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:28:50.313544 11072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:50.313606 11074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:28:50.314036 11071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:28:50.314314 11066 server_base.cc:1034] running on GCE node
I20260619 06:28:50.314946 11066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:28:50.315779 11066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:28:50.316988 11066 hybrid_clock.cc:648] HybridClock initialized: now 1781850530316943 us; error 43 us; skew 500 ppm
I20260619 06:28:50.317307 11066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:28:50 dist-test-slave-4t1f krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:28:50.320225 11066 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:28:50.321473 11066 webserver.cc:460] Webserver started at http://127.0.97.67:39983/ using document root <none> and password file <none>
I20260619 06:28:50.322190 11066 fs_manager.cc:359] Metadata directory not provided
I20260619 06:28:50.322283 11066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:28:50.322513 11066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:28:50.325032 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "fb4552c5ddc74bcba8586c710ebbade1"
format_stamp: "Formatted at 2026-06-19 06:28:50 on dist-test-slave-4t1f"
server_key: "37a05921cda89f6f7b3bcb811860f8bf"
server_key_iv: "b6bc29b66877df0b05f426eb69cfbdf4"
server_key_version: "encryptionkey@0"
I20260619 06:28:50.325752 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "fb4552c5ddc74bcba8586c710ebbade1"
format_stamp: "Formatted at 2026-06-19 06:28:50 on dist-test-slave-4t1f"
server_key: "37a05921cda89f6f7b3bcb811860f8bf"
server_key_iv: "b6bc29b66877df0b05f426eb69cfbdf4"
server_key_version: "encryptionkey@0"
I20260619 06:28:50.330395 11066 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 06:28:50.333814 11081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:50.335181 11066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:28:50.335305 11066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "fb4552c5ddc74bcba8586c710ebbade1"
format_stamp: "Formatted at 2026-06-19 06:28:50 on dist-test-slave-4t1f"
server_key: "37a05921cda89f6f7b3bcb811860f8bf"
server_key_iv: "b6bc29b66877df0b05f426eb69cfbdf4"
server_key_version: "encryptionkey@0"
I20260619 06:28:50.335462 11066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:28:50.368324 11066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:28:50.372534 11066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:28:50.372761 11066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:28:50.373688 11066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:28:50.374887 11066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:28:50.374972 11066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:50.375161 11066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:28:50.375226 11066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:28:50.407395 11066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44999
I20260619 06:28:50.407442 11193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44999 every 8 connection(s)
I20260619 06:28:50.408560 11066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:28:50.415679 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 11066
I20260619 06:28:50.415879 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:28:50.416431 389 external_mini_cluster.cc:1428] Setting key 1d8a730be782b5455111e1ab324ad295
Jun 19 06:28:50 dist-test-slave-4t1f krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:28:50.427606 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.410882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59969 (local address 127.0.97.126:46777)
0619 06:28:50.411095 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.411101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.412215 (+ 1114us) server_negotiation.cc:407] Connection header received
0619 06:28:50.413953 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.413957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.414022 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.414111 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:50.415460 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.416654 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.418504 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.419161 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.419302 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.422564 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:50.422569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:50.422572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:50.422599 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:50.425223 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.425844 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.425847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.425849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.425958 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.426300 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.426303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.426305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.426537 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:50.426642 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.427238 (+ 596us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.427378 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:28:50.428495 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.411371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46777 (local address 127.0.97.67:59969)
0619 06:28:50.412052 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.412094 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.413699 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.414292 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.414299 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.414761 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:28:50.415283 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.415298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.416801 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.416805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.418353 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.418366 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.419366 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.419391 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.419687 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.420281 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:28:50.420305 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:28:50.422310 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:28:50.425339 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:50.425345 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:50.425355 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:50.425742 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:50.426043 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:28:50.426046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:28:50.426049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:28:50.426202 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:28:50.426650 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:28:50.426662 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:28:50.426921 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.427161 (+ 240us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.427482 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":292,"threads_started":1}
I20260619 06:28:50.429806 11194 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46777
I20260619 06:28:50.430140 11194 heartbeater.cc:464] Registering TS with master...
I20260619 06:28:50.431006 11194 heartbeater.cc:511] Master 127.0.97.126:46777 requested a full tablet report, sending...
I20260619 06:28:50.432350 10927 heartbeater.cc:503] Master 127.0.97.126:46777 was elected leader, sending a full tablet report...
I20260619 06:28:50.432410 10738 ts_manager.cc:194] Registered new tserver with Master: fb4552c5ddc74bcba8586c710ebbade1 (127.0.97.67:44999)
I20260619 06:28:50.433457 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:59969
I20260619 06:28:50.443234 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:28:50.458705 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.447625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.0.97.126:46777)
0619 06:28:50.447802 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.447806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.448227 (+ 421us) server_negotiation.cc:407] Connection header received
0619 06:28:50.448349 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.448352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.448403 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.448478 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:50.449811 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.450484 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.451604 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.451987 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.452080 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.454149 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:50.454153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:50.454156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:50.454182 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:50.456584 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.457194 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.457197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.457199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.457309 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.457610 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.457612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.457614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.457873 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:50.458036 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.458350 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.458491 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:28:50.462318 10738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:28:50.469005 10738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:28:50.493782 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.480121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40845 (local address 127.0.0.1:45182)
0619 06:28:50.481679 (+ 1558us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.481726 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.481949 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.483222 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.483229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.483261 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:50.483784 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.483803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.486365 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.486375 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.489571 (+ 3196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.489593 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.492610 (+ 3017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.492616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.492905 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.492996 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.493155 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.493357 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"mutex_wait_us":80,"thread_start_us":10090,"threads_started":1}
I20260619 06:28:50.493782 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.480289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.0.97.65:40845)
0619 06:28:50.481007 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.481016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.482206 (+ 1190us) server_negotiation.cc:407] Connection header received
0619 06:28:50.482342 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.482354 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.482673 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.482884 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:50.484049 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.486148 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.490362 (+ 4214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.492077 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.492184 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.492437 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.493233 (+ 796us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.493417 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":619,"threads_started":1}
I20260619 06:28:50.498067 10862 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4375f67b5941db81044a0a86fa4787 (DEFAULT_TABLE table=test-table [id=e99d3609fee047aea0d3085844aaf780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:50.499076 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.480750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.0.97.66:44857)
0619 06:28:50.490251 (+ 9501us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.490259 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.490692 (+ 433us) server_negotiation.cc:407] Connection header received
0619 06:28:50.490900 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.490909 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.491167 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.491359 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:50.492360 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.494149 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.496580 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.497978 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.498126 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.498320 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.498537 (+ 217us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.498695 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9298,"thread_start_us":9130,"threads_started":1}
I20260619 06:28:50.499058 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.480161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44857 (local address 127.0.0.1:57550)
0619 06:28:50.490547 (+ 10386us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.490581 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.490727 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.491600 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.491606 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.491718 (+ 112us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:50.492142 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.492163 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.494299 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.494304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.496419 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.496432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.498187 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.498193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.498356 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.498390 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.498499 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.498622 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10208,"mutex_wait_us":1,"thread_start_us":2639,"threads_started":1}
I20260619 06:28:50.499557 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.481778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44999 (local address 127.0.0.1:34862)
0619 06:28:50.491890 (+ 10112us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.491906 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.492032 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.492717 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.492721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.492745 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:50.493249 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.493261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.495150 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.495157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.496958 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.496968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.498518 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.498523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.498683 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.498724 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.498809 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.498926 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9989,"mutex_wait_us":5,"thread_start_us":9838,"threads_started":1}
I20260619 06:28:50.499675 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.482537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34862 (local address 127.0.97.67:44999)
0619 06:28:50.491418 (+ 8881us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.491430 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.492174 (+ 744us) server_negotiation.cc:407] Connection header received
0619 06:28:50.492267 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.492274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.492507 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.492735 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:50.493438 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.494969 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.497144 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.498014 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.498175 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.498386 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.499159 (+ 773us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.499330 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8685,"thread_start_us":8523,"threads_started":1}
I20260619 06:28:50.499945 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4375f67b5941db81044a0a86fa4787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:50.502708 11129 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4375f67b5941db81044a0a86fa4787 (DEFAULT_TABLE table=test-table [id=e99d3609fee047aea0d3085844aaf780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:50.503098 10996 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4375f67b5941db81044a0a86fa4787 (DEFAULT_TABLE table=test-table [id=e99d3609fee047aea0d3085844aaf780]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:28:50.503845 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4375f67b5941db81044a0a86fa4787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:50.504374 10996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4375f67b5941db81044a0a86fa4787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:28:50.514557 11212 tablet_bootstrap.cc:492] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Bootstrap starting.
I20260619 06:28:50.515472 11213 tablet_bootstrap.cc:492] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Bootstrap starting.
I20260619 06:28:50.519093 11212 tablet_bootstrap.cc:654] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:50.519109 11213 tablet_bootstrap.cc:654] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:50.519258 11214 tablet_bootstrap.cc:492] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Bootstrap starting.
I20260619 06:28:50.520365 11213 log.cc:824] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:50.520388 11212 log.cc:824] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:50.522598 11214 tablet_bootstrap.cc:654] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Neither blocks nor log segments found. Creating new log.
I20260619 06:28:50.523196 11213 tablet_bootstrap.cc:492] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: No bootstrap required, opened a new log
I20260619 06:28:50.523449 11213 ts_tablet_manager.cc:1397] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 06:28:50.523612 11212 tablet_bootstrap.cc:492] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: No bootstrap required, opened a new log
I20260619 06:28:50.523820 11214 log.cc:824] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:28:50.523885 11212 ts_tablet_manager.cc:1397] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260619 06:28:50.526710 11214 tablet_bootstrap.cc:492] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: No bootstrap required, opened a new log
I20260619 06:28:50.526943 11214 ts_tablet_manager.cc:1397] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 06:28:50.531478 11213 raft_consensus.cc:348] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.531831 11213 raft_consensus.cc:374] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:50.531926 11213 raft_consensus.cc:729] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 145f9cc5601d41b98393b0dea489af98, State: Initialized, Role: FOLLOWER
I20260619 06:28:50.532580 11213 consensus_queue.cc:260] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.534116 11213 ts_tablet_manager.cc:1428] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260619 06:28:50.534991 11212 raft_consensus.cc:348] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.535415 11212 raft_consensus.cc:374] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:50.535507 11212 raft_consensus.cc:729] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb4552c5ddc74bcba8586c710ebbade1, State: Initialized, Role: FOLLOWER
I20260619 06:28:50.536043 11212 consensus_queue.cc:260] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.537176 11194 heartbeater.cc:503] Master 127.0.97.126:46777 was elected leader, sending a full tablet report...
I20260619 06:28:50.537099 11214 raft_consensus.cc:348] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.537511 11214 raft_consensus.cc:374] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:28:50.537602 11214 raft_consensus.cc:729] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683a8a5dbfa543d391e447a7a5b7ff6b, State: Initialized, Role: FOLLOWER
I20260619 06:28:50.537786 11212 ts_tablet_manager.cc:1428] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260619 06:28:50.538106 11214 consensus_queue.cc:260] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.540019 11061 heartbeater.cc:503] Master 127.0.97.126:46777 was elected leader, sending a full tablet report...
I20260619 06:28:50.541392 11214 ts_tablet_manager.cc:1428] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260619 06:28:50.650578 11219 raft_consensus.cc:482] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260619 06:28:50.650640 11062 tablet.cc:2369] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:28:50.650895 11219 raft_consensus.cc:504] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.652531 11219 leader_election.cc:288] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 145f9cc5601d41b98393b0dea489af98 (127.0.97.65:40845), 683a8a5dbfa543d391e447a7a5b7ff6b (127.0.97.66:44857)
W20260619 06:28:50.655102 10928 tablet.cc:2369] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:28:50.655485 11218 raft_consensus.cc:482] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:28:50.655795 11218 raft_consensus.cc:504] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.657614 11218 leader_election.cc:288] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 683a8a5dbfa543d391e447a7a5b7ff6b (127.0.97.66:44857), fb4552c5ddc74bcba8586c710ebbade1 (127.0.97.67:44999)
W20260619 06:28:50.660017 11195 tablet.cc:2369] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:28:50.660162 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.652893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:49599 (local address 127.0.97.65:40845)
0619 06:28:50.653133 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.653141 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.653733 (+ 592us) server_negotiation.cc:407] Connection header received
0619 06:28:50.653892 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.653897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.653967 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.654061 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:50.654664 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.655759 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.657730 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.659067 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.659191 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.659292 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.659814 (+ 522us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.659933 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:28:50.660441 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.653301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60391 (local address 127.0.97.66:44857)
0619 06:28:50.653494 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.653499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.653518 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:28:50.653601 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.653605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.653710 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.653804 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:50.654203 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.655075 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.657412 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.658675 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.658811 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.660038 (+ 1227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.660088 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.660217 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:28:50.660804 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.652881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:40845 (local address 127.0.97.67:49599)
0619 06:28:50.653608 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.653624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.653764 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.654121 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.654124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.654162 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:50.654513 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.654526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.655937 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.655947 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.657557 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.657570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.659490 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.659496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.659654 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.659685 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.660288 (+ 603us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.660404 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":47,"thread_start_us":407,"threads_started":1}
I20260619 06:28:50.661252 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.652858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44857 (local address 127.0.97.67:60391)
0619 06:28:50.653113 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.653135 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.653299 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.653854 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.653859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.653885 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:50.654073 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.654084 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.655222 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.655227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.657238 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.657252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.659317 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.659322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.659543 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.659581 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.659672 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.659814 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":1}
I20260619 06:28:50.662364 11016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4375f67b5941db81044a0a86fa4787" candidate_uuid: "fb4552c5ddc74bcba8586c710ebbade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" is_pre_election: true
I20260619 06:28:50.662873 11016 raft_consensus.cc:2454] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4552c5ddc74bcba8586c710ebbade1 in term 0.
I20260619 06:28:50.663309 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4375f67b5941db81044a0a86fa4787" candidate_uuid: "fb4552c5ddc74bcba8586c710ebbade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145f9cc5601d41b98393b0dea489af98" is_pre_election: true
I20260619 06:28:50.663564 11083 leader_election.cc:302] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 683a8a5dbfa543d391e447a7a5b7ff6b, fb4552c5ddc74bcba8586c710ebbade1; no voters:
I20260619 06:28:50.663735 10882 raft_consensus.cc:2454] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb4552c5ddc74bcba8586c710ebbade1 in term 0.
I20260619 06:28:50.663980 11219 raft_consensus.cc:2781] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:28:50.664150 11219 raft_consensus.cc:482] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:28:50.664264 11219 raft_consensus.cc:3037] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:50.667567 11219 raft_consensus.cc:504] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.668148 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.658129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:53139 (local address 127.0.97.66:44857)
0619 06:28:50.661135 (+ 3006us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.661140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.661908 (+ 768us) server_negotiation.cc:407] Connection header received
0619 06:28:50.662164 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.662169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.662239 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.662328 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:50.662966 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.664082 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.665956 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.667173 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.667310 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.667430 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.667765 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.667889 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2883,"thread_start_us":3141,"threads_started":1}
I20260619 06:28:50.668265 11223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.657725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44857 (local address 127.0.97.65:53139)
0619 06:28:50.661806 (+ 4081us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.661825 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.662022 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.662366 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.662374 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.662399 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:50.662812 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.662828 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.664245 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.664250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.665799 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.665811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.667383 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.667392 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.667564 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.667598 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.667744 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.667873 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3952,"thread_start_us":4707,"threads_started":1}
I20260619 06:28:50.668707 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4375f67b5941db81044a0a86fa4787" candidate_uuid: "fb4552c5ddc74bcba8586c710ebbade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "145f9cc5601d41b98393b0dea489af98"
I20260619 06:28:50.668921 10882 raft_consensus.cc:3037] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:50.668923 11016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4375f67b5941db81044a0a86fa4787" candidate_uuid: "145f9cc5601d41b98393b0dea489af98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" is_pre_election: true
I20260619 06:28:50.669139 11016 raft_consensus.cc:2454] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 145f9cc5601d41b98393b0dea489af98 in term 0.
I20260619 06:28:50.669152 11219 leader_election.cc:288] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [CANDIDATE]: Term 1 election: Requested vote from peers 145f9cc5601d41b98393b0dea489af98 (127.0.97.65:40845), 683a8a5dbfa543d391e447a7a5b7ff6b (127.0.97.66:44857)
I20260619 06:28:50.669673 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.659143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:34105 (local address 127.0.97.67:44999)
0619 06:28:50.659341 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.659349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.662290 (+ 2941us) server_negotiation.cc:407] Connection header received
0619 06:28:50.662474 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.662482 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.662555 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.662614 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:28:50.664581 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.665670 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.667448 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.668684 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.668832 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.668932 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.669328 (+ 396us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.669460 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 06:28:50.669662 10816 leader_election.cc:302] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145f9cc5601d41b98393b0dea489af98, 683a8a5dbfa543d391e447a7a5b7ff6b; no voters:
I20260619 06:28:50.670296 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.657992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44999 (local address 127.0.97.65:34105)
0619 06:28:50.662200 (+ 4208us) negotiation.cc:107] Waiting for socket to connect
0619 06:28:50.662213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:28:50.662338 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:28:50.662867 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:28:50.662874 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:28:50.662903 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:28:50.664369 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.664385 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.665813 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.665818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.667250 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:28:50.667265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.668898 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.668907 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:28:50.669102 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.669139 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:28:50.669818 (+ 679us) client_negotiation.cc:240] Negotiation successful
0619 06:28:50.669960 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4091,"thread_start_us":3877,"threads_started":1}
I20260619 06:28:50.670882 11016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4375f67b5941db81044a0a86fa4787" candidate_uuid: "fb4552c5ddc74bcba8586c710ebbade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b"
I20260619 06:28:50.671098 11016 raft_consensus.cc:3037] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:28:50.671595 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c4375f67b5941db81044a0a86fa4787" candidate_uuid: "145f9cc5601d41b98393b0dea489af98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb4552c5ddc74bcba8586c710ebbade1" is_pre_election: true
I20260619 06:28:50.672070 11149 raft_consensus.cc:2379] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 145f9cc5601d41b98393b0dea489af98 in current term 1: Already voted for candidate fb4552c5ddc74bcba8586c710ebbade1 in this term.
I20260619 06:28:50.672099 10882 raft_consensus.cc:2454] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4552c5ddc74bcba8586c710ebbade1 in term 1.
I20260619 06:28:50.672354 11218 raft_consensus.cc:2741] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 06:28:50.672583 11082 leader_election.cc:302] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 145f9cc5601d41b98393b0dea489af98, fb4552c5ddc74bcba8586c710ebbade1; no voters:
I20260619 06:28:50.672946 11219 raft_consensus.cc:2781] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:28:50.673465 11219 raft_consensus.cc:686] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 1 LEADER]: Becoming Leader. State: Replica: fb4552c5ddc74bcba8586c710ebbade1, State: Running, Role: LEADER
I20260619 06:28:50.674063 11016 raft_consensus.cc:2454] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb4552c5ddc74bcba8586c710ebbade1 in term 1.
I20260619 06:28:50.674024 11219 consensus_queue.cc:237] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } }
I20260619 06:28:50.678668 10738 catalog_manager.cc:5495] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb4552c5ddc74bcba8586c710ebbade1 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "fb4552c5ddc74bcba8586c710ebbade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb4552c5ddc74bcba8586c710ebbade1" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44999 } health_report { overall_health: HEALTHY } } }
I20260619 06:28:50.734948 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.729964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34874 (local address 127.0.97.67:44999)
0619 06:28:50.730125 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.730130 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.730147 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:28:50.730328 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.730331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.730428 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.730545 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:28:50.730902 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.731610 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.732997 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.733430 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.733520 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.733925 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:28:50.734397 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:28:50.734501 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.734571 (+ 70us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.734627 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:28:50.742652 10882 raft_consensus.cc:1261] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 1 FOLLOWER]: Refusing update from remote peer fb4552c5ddc74bcba8586c710ebbade1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:50.742758 11016 raft_consensus.cc:1261] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 1 FOLLOWER]: Refusing update from remote peer fb4552c5ddc74bcba8586c710ebbade1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:28:50.743580 11225 consensus_queue.cc:1035] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "145f9cc5601d41b98393b0dea489af98" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:50.743948 11219 consensus_queue.cc:1035] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "683a8a5dbfa543d391e447a7a5b7ff6b" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:28:50.766733 11230 mvcc.cc:205] Tried to move back new op lower bound from 7298459773912629248 to 7298459773645574144. Current Snapshot: MvccSnapshot[applied={T|T < 7298459773912629248}]
I20260619 06:28:50.768290 10738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:50.768573 10738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:28:50.771682 10738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75eb2ddc0f6b4b5a8b90b89b93d45853: Sending DeleteTablet for 3 replicas of tablet 8c4375f67b5941db81044a0a86fa4787
I20260619 06:28:50.773015 11129 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4375f67b5941db81044a0a86fa4787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:50 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:34862
I20260619 06:28:50.773032 10996 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4375f67b5941db81044a0a86fa4787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:50 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:57550
I20260619 06:28:50.773689 10862 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c4375f67b5941db81044a0a86fa4787 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:28:50 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45182
I20260619 06:28:50.773893 11238 tablet_replica.cc:330] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: stopping tablet replica
I20260619 06:28:50.774272 11238 raft_consensus.cc:2229] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:50.774549 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10799
I20260619 06:28:50.774562 11239 tablet_replica.cc:330] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: stopping tablet replica
I20260619 06:28:50.774713 11238 raft_consensus.cc:2258] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:50.774931 11239 raft_consensus.cc:2229] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:28:50.775279 11239 raft_consensus.cc:2258] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:50.777488 11239 ts_tablet_manager.cc:1905] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:50.777969 11238 ts_tablet_manager.cc:1905] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:28:50.784544 11239 ts_tablet_manager.cc:1918] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:50.784637 11238 ts_tablet_manager.cc:1918] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:50.784706 11239 log.cc:1196] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/8c4375f67b5941db81044a0a86fa4787
I20260619 06:28:50.784780 11238 log.cc:1196] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/8c4375f67b5941db81044a0a86fa4787
I20260619 06:28:50.785449 11238 ts_tablet_manager.cc:1939] T 8c4375f67b5941db81044a0a86fa4787 P 683a8a5dbfa543d391e447a7a5b7ff6b: Deleting consensus metadata
I20260619 06:28:50.785467 11239 ts_tablet_manager.cc:1939] T 8c4375f67b5941db81044a0a86fa4787 P 145f9cc5601d41b98393b0dea489af98: Deleting consensus metadata
I20260619 06:28:50.787302 10723 catalog_manager.cc:4841] TS 683a8a5dbfa543d391e447a7a5b7ff6b (127.0.97.66:44857): tablet 8c4375f67b5941db81044a0a86fa4787 (table test-table [id=e99d3609fee047aea0d3085844aaf780]) successfully deleted
I20260619 06:28:50.787328 10722 catalog_manager.cc:4841] TS 145f9cc5601d41b98393b0dea489af98 (127.0.97.65:40845): tablet 8c4375f67b5941db81044a0a86fa4787 (table test-table [id=e99d3609fee047aea0d3085844aaf780]) successfully deleted
I20260619 06:28:50.790304 11237 tablet_replica.cc:330] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: stopping tablet replica
I20260619 06:28:50.790755 11237 raft_consensus.cc:2229] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:28:50.791417 11237 raft_consensus.cc:2258] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:28:50.794754 11237 ts_tablet_manager.cc:1905] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Deleting tablet data with delete state TABLET_DATA_DELETED
Jun 19 06:28:50 dist-test-slave-4t1f krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:28:50.801440 11237 ts_tablet_manager.cc:1918] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:28:50.801614 11237 log.cc:1196] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/8c4375f67b5941db81044a0a86fa4787
I20260619 06:28:50.802186 11237 ts_tablet_manager.cc:1939] T 8c4375f67b5941db81044a0a86fa4787 P fb4552c5ddc74bcba8586c710ebbade1: Deleting consensus metadata
I20260619 06:28:50.804036 10725 catalog_manager.cc:4841] TS fb4552c5ddc74bcba8586c710ebbade1 (127.0.97.67:44999): tablet 8c4375f67b5941db81044a0a86fa4787 (table test-table [id=e99d3609fee047aea0d3085844aaf780]) successfully deleted
I20260619 06:28:50.807011 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.776360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45184 (local address 127.0.97.65:40845)
0619 06:28:50.776526 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.776533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.789621 (+ 13088us) server_negotiation.cc:407] Connection header received
0619 06:28:50.789834 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.789839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.789933 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.790031 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:50.791701 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.792793 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.794595 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.795223 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.795347 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.800270 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:50.800285 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:50.800302 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:50.800345 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:50.804141 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.804924 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.804933 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.804937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.805141 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.805492 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.805497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.805501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.805945 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:50.806258 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.806412 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.806636 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:28:50.807605 10922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:50.942397 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10799
I20260619 06:28:50.965786 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10933
Jun 19 06:28:50 dist-test-slave-4t1f krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:28:50.981940 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:50.966607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.0.97.66:44857)
0619 06:28:50.966777 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:50.966782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:50.966857 (+ 75us) server_negotiation.cc:407] Connection header received
0619 06:28:50.967031 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:50.967035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:50.967107 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:50.967198 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:50.968640 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.969506 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.971286 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:50.971654 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:50.971743 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:50.976573 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:50.976585 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:50.976600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:50.976629 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:50.979421 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.980175 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.980182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.980184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.980295 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:50.980641 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:50.980645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:50.980648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:50.980962 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:50.981272 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:50.981517 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 06:28:50.981662 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:28:50.982563 11056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:51.071906 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10933
I20260619 06:28:51.094308 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 11066
Jun 19 06:28:51 dist-test-slave-4t1f krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:28:51.111418 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:28:51.095084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.0.97.67:44999)
0619 06:28:51.095233 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:28:51.095237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:28:51.095309 (+ 72us) server_negotiation.cc:407] Connection header received
0619 06:28:51.095487 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:28:51.095491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:28:51.095571 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:28:51.095663 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:28:51.097354 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:51.098106 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:51.099861 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:28:51.100282 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:28:51.100371 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:28:51.105623 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:28:51.105634 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:28:51.105654 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:28:51.105686 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:28:51.108688 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:51.109384 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:51.109392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:51.109394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:51.109529 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:28:51.109899 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:28:51.109902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:28:51.109904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:28:51.110259 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:28:51.110553 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0619 06:28:51.110867 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 06:28:51.111117 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 06:28:51.112164 11189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:51.229578 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 11066
I20260619 06:28:51.248226 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 10706
I20260619 06:28:51.249279 10768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:28:51.354799 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 10706
2026-06-19T06:28:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4276 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:28:51 dist-test-slave-4t1f krb5kdc[11247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:28:51 dist-test-slave-4t1f krb5kdc[11247](info): set up 2 sockets
Jun 19 06:28:51 dist-test-slave-4t1f krb5kdc[11247](info): commencing operation
krb5kdc: starting...
W20260619 06:28:52.437958 389 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.005s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:28:52 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:28:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:28:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
I20260619 06:28:52.735385 389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWesBqs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/postgres -l logfile start
2026-06-19 06:28:55.051 UTC [11277] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:28:55.051 UTC [11277] LOG: listening on IPv4 address "127.0.97.84", port 52471
2026-06-19 06:28:55.054 UTC [11277] LOG: listening on Unix socket "/tmp/.s.PGSQL.52471"
2026-06-19 06:28:55.077 UTC [11280] LOG: database system was shut down at 2026-06-19 06:28:53 UTC
2026-06-19 06:28:55.080 UTC [11277] LOG: database system is ready to accept connections
I20260619 06:28:56.045499 389 mini_postgres.cc:96] Postgres bound to 52471
2026-06-19 06:28:56.062 UTC [11288] FATAL: database "slave" does not exist
127.0.97.84:52471 - accepting connections
I20260619 06:28:56.063374 389 mini_ranger.cc:161] Starting Ranger...
I20260619 06:28:56.091099 389 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:28:56.535096 389 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:28:56.535280 389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-admin
2026-06-19 06:28:56,819 [I] DB FLAVOR :POSTGRES
2026-06-19 06:28:56,820 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:28:56,820 [I] Checking connection..
2026-06-19 06:28:56,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:28:57,084 [I] Checking connection passed.
2026-06-19 06:28:57,085 [I] --------- Verifying version history table ---------
2026-06-19 06:28:57,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:28:57,311 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:28:57,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:28:57,560 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:28:57,560 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:28:57,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:28:57.811 UTC [11352] WARNING: there is no transaction in progress
2026-06-19 06:28:57,817 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:28:57,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:28:58,050 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:28:58,051 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:28:58,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:28:58,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:28:58,549 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4t1f.c.gcp-upstream.internal','N') ;"
2026-06-19 06:28:58,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:28:59,022 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:28:59,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:28:59,275 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:28:59,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:28:59,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:28:59,757 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:28:59,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:29:00.670 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.685 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.698 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.712 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.729 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.744 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.794 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.801 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.814 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.823 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.832 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.845 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.854 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.860 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.866 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:00.872 UTC [11460] WARNING: there is no transaction in progress
2026-06-19 06:29:01,444 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:29:01,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:29:01,676 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:29:01,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:29:01,915 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:29:01,915 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:29:02,178 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:29:02,178 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:29:02,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:29:02,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4t1f.c.gcp-upstream.internal';"
2026-06-19 06:29:02,868 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:29:02,869 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:29:03,127 [I] DB_PATCHES have already been applied
I20260619 06:29:03.133706 389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWesBqs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:29:03.133818 389 mini_ranger.cc:193] Using host: 127.0.97.84
I20260619 06:29:03.147922 389 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.84:49547
Jun 19, 2026 6:29:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:29:03 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:29:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:29:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:29:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:29:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:29:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:29:04.282004 389 mini_ranger.cc:160] Time spent starting Ranger: real 8.219s user 0.003s sys 0.033s
Jun 19, 2026 6:29:23 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:29:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:29:23 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:29:23 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:29:24 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:29:25.406092 389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:29:25 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850565, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:29:25.557168 389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:29:25.577006 389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:29:26.348836 389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:29:26.348966 389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms
I20260619 06:29:26.349138 389 mini_ranger_kms.cc:227] Using postgres at 127.0.97.84:52471
2026-06-19 06:29:26,773 [I] DB FLAVOR :POSTGRES
2026-06-19 06:29:26,774 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:29:26,774 [I] Checking connection
2026-06-19 06:29:26,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:29:27,005 [I] connection success
2026-06-19 06:29:27,005 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:29:27,005 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:29:27,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:29:27,225 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:29:27,225 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:29:27,225 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:29:27,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:52471/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:29:27,481 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:29:27.928613 389 mini_ranger_kms.cc:310] Created kms service
I20260619 06:29:28.129163 389 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:29:28.171705 389 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:29:28 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850565, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:29:28.545508 389 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:29:28.545840 389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:29:28.545884 389 mini_ranger_kms.cc:240] Using host: 127.0.97.84
I20260619 06:29:28.566170 389 mini_ranger_kms.cc:276] Ranger KMS PID: 11640
I20260619 06:29:28.566324 389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.84:36339
Jun 19, 2026 6:29:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:29:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:29:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:29:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:29:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:29:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 06:29:30 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850570, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:29:30.881660 389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.325s user 0.145s sys 0.293s
I20260619 06:29:30.881991 389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:29:30.882215 389 mini_ranger_kms.cc:401] 127.0.97.84:36339/kms/v1/keys
Jun 19 06:29:31 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850570, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 19 06:29:31 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850570, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 19 06:29:33 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:29:33.981467 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42583
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:42583
--ranger_config_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:36339/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:29:34.311610 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:29:34.311933 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:29:34.312019 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:29:34.319269 11691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:29:34.319363 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:29:34.319404 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:29:34.319448 11691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:29:34.319486 11691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:29:34.329085 11691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.84:36339/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:42583
--ranger_config_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:42583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:29:34.330260 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:29:34.331456 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:29:34.343339 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:34.343377 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:34.343406 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:29:34.344221 11691 server_base.cc:1034] running on GCE node
I20260619 06:29:34.344846 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:29:34.346233 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:29:34.347424 11691 hybrid_clock.cc:648] HybridClock initialized: now 1781850574347382 us; error 35 us; skew 500 ppm
I20260619 06:29:34.347760 11691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:29:34 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:29:34.354920 11691 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:29:34.356343 11691 webserver.cc:460] Webserver started at http://127.0.97.126:39935/ using document root <none> and password file <none>
I20260619 06:29:34.356966 11691 fs_manager.cc:359] Metadata directory not provided
I20260619 06:29:34.357120 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:29:34.357421 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:29:34 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:29:34.404770 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "62e2cda45ece4f588ead791114604341"
format_stamp: "Formatted at 2026-06-19 06:29:34 on dist-test-slave-4t1f"
server_key: "af15a22e6f24090a12c33e7b2f9934b8"
server_key_iv: "bb6d9f63fe89c426a9d4357fa29c353e"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:34.405695 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "62e2cda45ece4f588ead791114604341"
format_stamp: "Formatted at 2026-06-19 06:29:34 on dist-test-slave-4t1f"
server_key: "af15a22e6f24090a12c33e7b2f9934b8"
server_key_iv: "bb6d9f63fe89c426a9d4357fa29c353e"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:34.411142 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:29:34.430639 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:34.432377 11691 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:29:34.432578 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "62e2cda45ece4f588ead791114604341"
format_stamp: "Formatted at 2026-06-19 06:29:34 on dist-test-slave-4t1f"
server_key: "af15a22e6f24090a12c33e7b2f9934b8"
server_key_iv: "bb6d9f63fe89c426a9d4357fa29c353e"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:34.432828 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:29:34.467903 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:29:34.471383 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:29:34.471639 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:29:34.490754 11691 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:42583
I20260619 06:29:34.490774 11757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:42583 every 8 connection(s)
I20260619 06:29:34.492117 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:29:34.495772 11758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:29:34.498188 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 11691
I20260619 06:29:34.498397 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:29:34.499015 389 external_mini_cluster.cc:1428] Setting key 853f8804450e232038e9145105b31e92
I20260619 06:29:34.508737 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341: Bootstrap starting.
I20260619 06:29:34.512344 11758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341: Neither blocks nor log segments found. Creating new log.
I20260619 06:29:34.513891 11758 log.cc:824] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:29:34 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:29:34.517359 11758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341: No bootstrap required, opened a new log
I20260619 06:29:34.523314 11761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:34.501393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.0.97.126:42583)
0619 06:29:34.502157 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:34.502172 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:34.502218 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:29:34.504552 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:34.504586 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:34.505113 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:34.505549 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:34.506649 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:34.508501 (+ 1852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:34.509857 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:34.510963 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:34.511128 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:34.515395 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:34.515406 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:34.515424 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:34.515461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:34.519126 (+ 3665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:34.519773 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:34.519782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:34.519790 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:34.520002 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:34.520328 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:34.520333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:34.520336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:34.521006 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:34.521435 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:34.521885 (+ 450us) server_negotiation.cc:299] Negotiation successful
0619 06:29:34.522289 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260619 06:29:34.526095 11758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2cda45ece4f588ead791114604341" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42583 } }
I20260619 06:29:34.526510 11758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:29:34.526634 11758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62e2cda45ece4f588ead791114604341, State: Initialized, Role: FOLLOWER
I20260619 06:29:34.527185 11758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2cda45ece4f588ead791114604341" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42583 } }
I20260619 06:29:34.527417 11758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:29:34.527958 11758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:29:34.528118 11758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:29:34.530606 11758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2cda45ece4f588ead791114604341" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42583 } }
I20260619 06:29:34.531059 11758 leader_election.cc:302] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62e2cda45ece4f588ead791114604341; no voters:
I20260619 06:29:34.531659 11758 leader_election.cc:288] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:29:34.531738 11763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:29:34.532538 11763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [term 1 LEADER]: Becoming Leader. State: Replica: 62e2cda45ece4f588ead791114604341, State: Running, Role: LEADER
I20260619 06:29:34.533210 11763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2cda45ece4f588ead791114604341" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42583 } }
I20260619 06:29:34.533849 11758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:29:34.536154 11764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62e2cda45ece4f588ead791114604341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2cda45ece4f588ead791114604341" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42583 } } }
I20260619 06:29:34.536115 11765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62e2cda45ece4f588ead791114604341. Latest consensus state: current_term: 1 leader_uuid: "62e2cda45ece4f588ead791114604341" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62e2cda45ece4f588ead791114604341" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 42583 } } }
I20260619 06:29:34.536561 11764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [sys.catalog]: This master's current role is: LEADER
I20260619 06:29:34.536561 11765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341 [sys.catalog]: This master's current role is: LEADER
I20260619 06:29:34.539008 11771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:29:34.544574 11771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:29:34.545260 11758 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:29:34.562116 11771 catalog_manager.cc:1347] Generated new cluster ID: c25e6764c0c14c73823e4110efc091bf
I20260619 06:29:34.562335 11771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:29:34.574549 11771 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:29:34.575760 11771 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:29:34.586723 11771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341: Generated new TSK 0
I20260619 06:29:34.587546 11771 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:29:35.318459 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:36339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42583
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:29:35 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:29:35.753376 11803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:29:35.753778 11803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:29:35.753985 11803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:29:35.765560 11803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:29:35.765746 11803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:29:35.765944 11803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:29:35.781963 11803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.84:36339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:29:35.783881 11803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:29:35.785668 11803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:29:35.808224 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:35.816763 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:35.818838 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:29:35.819226 11803 server_base.cc:1034] running on GCE node
I20260619 06:29:35.819974 11803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:29:35.820996 11803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:29:35.822240 11803 hybrid_clock.cc:648] HybridClock initialized: now 1781850575822183 us; error 54 us; skew 500 ppm
I20260619 06:29:35.822609 11803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:29:35 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:29:35.826198 11803 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:29:35.827885 11803 webserver.cc:460] Webserver started at http://127.0.97.65:33985/ using document root <none> and password file <none>
I20260619 06:29:35.828567 11803 fs_manager.cc:359] Metadata directory not provided
I20260619 06:29:35.828696 11803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:29:35.829003 11803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:29:35 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:29:35.856695 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "931d9ec3f51943449ccd90d51a37fe0e"
format_stamp: "Formatted at 2026-06-19 06:29:35 on dist-test-slave-4t1f"
server_key: "0252f77b71004bb88ed8bdb2982681a1"
server_key_iv: "0dd282c9ebb9c1018a33907c897bc34b"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:35.858012 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "931d9ec3f51943449ccd90d51a37fe0e"
format_stamp: "Formatted at 2026-06-19 06:29:35 on dist-test-slave-4t1f"
server_key: "0252f77b71004bb88ed8bdb2982681a1"
server_key_iv: "0dd282c9ebb9c1018a33907c897bc34b"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:35.874930 11803 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.008s sys 0.008s
I20260619 06:29:35.908743 11823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:35.910807 11803 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.009s sys 0.009s
I20260619 06:29:35.911082 11803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "931d9ec3f51943449ccd90d51a37fe0e"
format_stamp: "Formatted at 2026-06-19 06:29:35 on dist-test-slave-4t1f"
server_key: "0252f77b71004bb88ed8bdb2982681a1"
server_key_iv: "0dd282c9ebb9c1018a33907c897bc34b"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:35.911458 11803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:29:35.973673 11803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:29:35.978943 11803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:29:35.979264 11803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:29:35.980346 11803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:29:35.989609 11803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:29:35.989732 11803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:35.989915 11803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:29:35.989991 11803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:29:36 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:29:36.063403 11803 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:34713
I20260619 06:29:36.063453 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:34713 every 8 connection(s)
I20260619 06:29:36.065555 11803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:29:36.070355 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 11803
I20260619 06:29:36.070552 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:29:36.071319 389 external_mini_cluster.cc:1428] Setting key 2878dd515b2a6192a4f29798b20cab8b
Jun 19 06:29:36 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:29:36.109995 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:36.072400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52021 (local address 127.0.97.126:42583)
0619 06:29:36.074445 (+ 2045us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:36.074451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:36.074470 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:29:36.078035 (+ 3565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:36.078041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:36.078124 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:36.078230 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:36.089482 (+ 11252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:36.090607 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:36.093518 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:36.094130 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:36.094229 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:36.102526 (+ 8297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:36.102533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:36.102538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:36.102589 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:36.106387 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:36.107259 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:36.107264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:36.107267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:36.107429 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:36.107920 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:36.107925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:36.107929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:36.108243 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:36.108358 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:36.109406 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0619 06:29:36.109613 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1921,"thread_start_us":4245,"threads_started":1}
I20260619 06:29:36.112262 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:36.071615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42583 (local address 127.0.97.65:52021)
0619 06:29:36.073170 (+ 1555us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:36.073231 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:36.076355 (+ 3124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:36.087495 (+ 11140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:36.087509 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:36.088220 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:29:36.089241 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:36.089278 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:36.090849 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:36.090854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:36.093336 (+ 2482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:36.093356 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:36.094572 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:36.094580 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:36.094862 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:36.097256 (+ 2394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:29:36.097288 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:29:36.102268 (+ 4980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:29:36.106562 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:29:36.106571 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:29:36.106589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:29:36.107127 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:29:36.107579 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:29:36.107583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:29:36.107586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:29:36.107800 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:29:36.108460 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:29:36.108468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:29:36.108920 (+ 452us) client_negotiation.cc:769] Sending connection context
0619 06:29:36.110385 (+ 1465us) client_negotiation.cc:240] Negotiation successful
0619 06:29:36.110906 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":981,"threads_started":1}
I20260619 06:29:36.114130 11936 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42583
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
I20260619 06:29:36.114434 11936 heartbeater.cc:464] Registering TS with master...
I20260619 06:29:36.115198 11936 heartbeater.cc:511] Master 127.0.97.126:42583 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
I20260619 06:29:36.117591 11723 ts_manager.cc:194] Registered new tserver with Master: 931d9ec3f51943449ccd90d51a37fe0e (127.0.97.65:34713)
I20260619 06:29:36.119413 11723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:52021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:29:36.152639 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:36339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42583
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:29:36 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260619 06:29:36.633857 11942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:29:36.634384 11942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:29:36.634603 11942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:29:36.646248 11942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:29:36.646452 11942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:29:36.646685 11942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:29:36.662832 11942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.84:36339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:29:36.664615 11942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:29:36.666291 11942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:29:36.701802 11950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:36.705521 11949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:36.717773 11952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:29:36.721756 11942 server_base.cc:1034] running on GCE node
I20260619 06:29:36.722618 11942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:29:36.732234 11942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:29:36.733685 11942 hybrid_clock.cc:648] HybridClock initialized: now 1781850576733569 us; error 112 us; skew 500 ppm
I20260619 06:29:36.734133 11942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:29:36 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:29:36.738799 11942 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:29:36.759333 11942 webserver.cc:460] Webserver started at http://127.0.97.66:42731/ using document root <none> and password file <none>
I20260619 06:29:36.760414 11942 fs_manager.cc:359] Metadata directory not provided
I20260619 06:29:36.760624 11942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:29:36.761036 11942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:29:36 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:29:36.792299 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "225183533ca2413da4335e8ab8d0d318"
format_stamp: "Formatted at 2026-06-19 06:29:36 on dist-test-slave-4t1f"
server_key: "dfbf46492fde723bd7429687d9a5997f"
server_key_iv: "141f19cddb7ffaff92b42e3b69131819"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:36.793805 11942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "225183533ca2413da4335e8ab8d0d318"
format_stamp: "Formatted at 2026-06-19 06:29:36 on dist-test-slave-4t1f"
server_key: "dfbf46492fde723bd7429687d9a5997f"
server_key_iv: "141f19cddb7ffaff92b42e3b69131819"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:36.804543 11942 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260619 06:29:36.837926 11966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:36.843173 11942 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.009s sys 0.007s
I20260619 06:29:36.843385 11942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "225183533ca2413da4335e8ab8d0d318"
format_stamp: "Formatted at 2026-06-19 06:29:36 on dist-test-slave-4t1f"
server_key: "dfbf46492fde723bd7429687d9a5997f"
server_key_iv: "141f19cddb7ffaff92b42e3b69131819"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:36.843653 11942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:29:36.914108 11942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:29:36.919859 11942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:29:36.920351 11942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:29:36.933624 11942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:29:36.950110 11942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:29:36.950315 11942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 06:29:36.950500 11942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:29:36.950611 11942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:37.012749 11942 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:36763
I20260619 06:29:37.012773 12078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:36763 every 8 connection(s)
I20260619 06:29:37.014425 11942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:29:37.015393 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 11942
I20260619 06:29:37.015573 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:29:37.016125 389 external_mini_cluster.cc:1428] Setting key f5956c6305f45811fd68bcadf38fb355
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Jun 19 06:29:37 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:29:37.087865 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.017333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:49825 (local address 127.0.97.126:42583)
0619 06:29:37.032579 (+ 15246us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.032589 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.051418 (+ 18829us) server_negotiation.cc:407] Connection header received
0619 06:29:37.054632 (+ 3214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.054637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.054726 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.054826 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:37.070480 (+ 15654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.071453 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.073951 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.074675 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.074788 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.080090 (+ 5302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:37.080100 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:37.080106 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:37.080153 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:37.083825 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.084790 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.084799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.084805 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.084951 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.085518 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.085525 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.085531 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.085936 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:37.086083 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.087077 (+ 994us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.087330 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15067,"thread_start_us":15254,"threads_started":1}
I20260619 06:29:37.090292 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.018019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42583 (local address 127.0.97.66:49825)
0619 06:29:37.051236 (+ 33217us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:37.051291 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:37.054298 (+ 3007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:37.068824 (+ 14526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:37.068837 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:37.069506 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:29:37.070265 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.070289 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.071636 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.071644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.073777 (+ 2133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.073795 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.075125 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.075133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.075471 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.076299 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:29:37.076333 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:29:37.079801 (+ 3468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:29:37.083991 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:29:37.084003 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:29:37.084024 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:29:37.084635 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:29:37.085119 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:29:37.085127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:29:37.085133 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:29:37.085382 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:29:37.086172 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:29:37.086182 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:29:37.086633 (+ 451us) client_negotiation.cc:769] Sending connection context
0619 06:29:37.088138 (+ 1505us) client_negotiation.cc:240] Negotiation successful
0619 06:29:37.088763 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32195,"thread_start_us":31961,"threads_started":1}
I20260619 06:29:37.091948 12079 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42583
I20260619 06:29:37.092265 12079 heartbeater.cc:464] Registering TS with master...
I20260619 06:29:37.093144 12079 heartbeater.cc:511] Master 127.0.97.126:42583 requested a full tablet report, sending...
I20260619 06:29:37.094758 11723 ts_manager.cc:194] Registered new tserver with Master: 225183533ca2413da4335e8ab8d0d318 (127.0.97.66:36763)
I20260619 06:29:37.095762 11723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:49825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:29:37.108448 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:36339/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42583
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:29:37.122989 11936 heartbeater.cc:503] Master 127.0.97.126:42583 was elected leader, sending a full tablet report...
W20260619 06:29:37.445240 12085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:29:37.445542 12085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:29:37.445710 12085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:29:37.452888 12085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:29:37.453028 12085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:29:37.453218 12085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:29:37.462539 12085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.84:36339/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:42583
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:29:37.463750 12085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:29:37.464995 12085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:29:37.477479 12093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:37.477530 12090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:29:37.477530 12091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:29:37.478380 12085 server_base.cc:1034] running on GCE node
I20260619 06:29:37.478966 12085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:29:37.479761 12085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:29:37.480947 12085 hybrid_clock.cc:648] HybridClock initialized: now 1781850577480903 us; error 38 us; skew 500 ppm
I20260619 06:29:37.481285 12085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:29:37 dist-test-slave-4t1f krb5kdc[11247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:29:37.484244 12085 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:29:37.486104 12085 webserver.cc:460] Webserver started at http://127.0.97.67:34825/ using document root <none> and password file <none>
I20260619 06:29:37.486719 12085 fs_manager.cc:359] Metadata directory not provided
I20260619 06:29:37.486858 12085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:29:37.487160 12085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:29:37 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:29:37.503247 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "d47d50d09db64cc0aeef214a0a51e443"
format_stamp: "Formatted at 2026-06-19 06:29:37 on dist-test-slave-4t1f"
server_key: "6624d14c199954407641cad03d044a5f"
server_key_iv: "717e1343e1e2406c3f2894b80c6e54c8"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:37.504043 12085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "d47d50d09db64cc0aeef214a0a51e443"
format_stamp: "Formatted at 2026-06-19 06:29:37 on dist-test-slave-4t1f"
server_key: "6624d14c199954407641cad03d044a5f"
server_key_iv: "717e1343e1e2406c3f2894b80c6e54c8"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:37.509354 12085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260619 06:29:37.524992 12100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:37.526628 12085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:29:37.526829 12085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "d47d50d09db64cc0aeef214a0a51e443"
format_stamp: "Formatted at 2026-06-19 06:29:37 on dist-test-slave-4t1f"
server_key: "6624d14c199954407641cad03d044a5f"
server_key_iv: "717e1343e1e2406c3f2894b80c6e54c8"
server_key_version: "kuduclusterkey@0"
I20260619 06:29:37.527064 12085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:29:37.540468 12085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:29:37.543955 12085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:29:37.544239 12085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:29:37.545114 12085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:29:37.546415 12085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:29:37.546510 12085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:37.546640 12085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:29:37.546705 12085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:29:37.580470 12085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:38199
I20260619 06:29:37.580519 12212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:38199 every 8 connection(s)
I20260619 06:29:37.581804 12085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:29:37.586933 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 12085
I20260619 06:29:37.587105 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:29:37.587532 389 external_mini_cluster.cc:1428] Setting key 4c0efb6633b37e6a5c6be0fa172e6075
Jun 19 06:29:37 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:29:37.600279 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.584619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59697 (local address 127.0.97.126:42583)
0619 06:29:37.584839 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.584845 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.585962 (+ 1117us) server_negotiation.cc:407] Connection header received
0619 06:29:37.587629 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.587636 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.587776 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.587891 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:37.589288 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.590009 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.591446 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.591850 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.591971 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.595095 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:37.595100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:37.595103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:37.595132 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:37.597475 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.598142 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.598152 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.598154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.598264 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.598670 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.598675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.598678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.598987 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:37.599145 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.599752 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.599963 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 06:29:37.601156 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.585117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:42583 (local address 127.0.97.67:59697)
0619 06:29:37.585810 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:37.585845 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:37.587381 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:37.588094 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:37.588101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:37.588524 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:29:37.589130 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.589147 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.590149 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.590154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.591302 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.591313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.592067 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.592074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.592292 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.592920 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:29:37.592940 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:29:37.594878 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:29:37.597612 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:29:37.597619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:29:37.597634 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:29:37.598017 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:29:37.598363 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:29:37.598369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:29:37.598373 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:29:37.598551 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:29:37.599119 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:29:37.599126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:29:37.599429 (+ 303us) client_negotiation.cc:769] Sending connection context
0619 06:29:37.599666 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 06:29:37.600106 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":290,"threads_started":1}
I20260619 06:29:37.602614 12213 heartbeater.cc:347] Connected to a master server at 127.0.97.126:42583
I20260619 06:29:37.603282 12213 heartbeater.cc:464] Registering TS with master...
I20260619 06:29:37.603946 12213 heartbeater.cc:511] Master 127.0.97.126:42583 requested a full tablet report, sending...
I20260619 06:29:37.605291 11722 ts_manager.cc:194] Registered new tserver with Master: d47d50d09db64cc0aeef214a0a51e443 (127.0.97.67:38199)
I20260619 06:29:37.606154 11722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:59697
I20260619 06:29:37.614605 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:29:37.631242 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.618942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.0.97.126:42583)
0619 06:29:37.619094 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.619098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.619549 (+ 451us) server_negotiation.cc:407] Connection header received
0619 06:29:37.619731 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.619736 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.619831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.619932 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:37.621326 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.622032 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.623502 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.623917 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.624043 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.626399 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:37.626409 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:37.626412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:37.626440 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:37.628587 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.629304 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.629310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.629314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.629434 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.629797 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.629802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.629805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.630120 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:37.630407 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.630698 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.630929 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:29:37.635375 11722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:29:37.642926 11722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:29:37.675872 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.654549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.97.67:38199)
0619 06:29:37.655795 (+ 1246us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.655803 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.656351 (+ 548us) server_negotiation.cc:407] Connection header received
0619 06:29:37.662611 (+ 6260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.662632 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.662955 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.663176 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:29:37.665164 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.666897 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.672547 (+ 5650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.674678 (+ 2131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.674836 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.675073 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.675363 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.675525 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":698,"threads_started":1}
I20260619 06:29:37.675889 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.654319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38199 (local address 127.0.0.1:38008)
0619 06:29:37.655981 (+ 1662us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:37.656016 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:37.656180 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:37.663602 (+ 7422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:37.663612 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:37.663651 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:29:37.664201 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.664217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.667196 (+ 2979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.667204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.672291 (+ 5087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.672310 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.674895 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.674901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.675132 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.675169 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:29:37.675317 (+ 148us) client_negotiation.cc:240] Negotiation successful
0619 06:29:37.675497 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1425,"thread_start_us":1505,"threads_started":1}
I20260619 06:29:37.679252 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.655766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34713 (local address 127.0.0.1:50474)
0619 06:29:37.670821 (+ 15055us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:37.670842 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:37.670949 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:37.671787 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:37.671793 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:37.671816 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:29:37.672313 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.672330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.674214 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.674226 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.676401 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.676417 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.678244 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.678251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.678534 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.678574 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:29:37.678694 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:29:37.678831 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14914,"mutex_wait_us":178,"thread_start_us":15257,"threads_started":1}
I20260619 06:29:37.679993 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.655771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50474 (local address 127.0.97.65:34713)
0619 06:29:37.670964 (+ 15193us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.670973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.671005 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:29:37.671104 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.671111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.671375 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.671570 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:29:37.672547 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.674094 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.676598 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.678036 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.678159 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.678348 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.679493 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.679654 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15022,"thread_start_us":16167,"threads_started":1}
I20260619 06:29:37.680830 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.656549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36763 (local address 127.0.0.1:56776)
0619 06:29:37.672574 (+ 16025us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:37.672602 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:37.672736 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:37.673772 (+ 1036us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:37.673777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:37.673804 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:29:37.674164 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.674176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.675988 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.675995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.678527 (+ 2532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.678537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.679979 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.679985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.680276 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.680308 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:29:37.680404 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:29:37.680515 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15897,"mutex_wait_us":38,"thread_start_us":15751,"threads_started":1}
I20260619 06:29:37.680972 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.656808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56776 (local address 127.0.97.66:36763)
0619 06:29:37.673249 (+ 16441us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.673257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.673281 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:29:37.673362 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.673369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.673578 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.673767 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:29:37.674346 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.675852 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.678678 (+ 2826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.679787 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.679929 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.680111 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.680466 (+ 355us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.680631 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16285,"thread_start_us":16170,"threads_started":1}
I20260619 06:29:37.680727 12148 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff8e9293c78424391997ee3eb0e4d95 (DEFAULT_TABLE table=test-table [id=807997729d644cdf947a29da15b3f09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:29:37.682344 12148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff8e9293c78424391997ee3eb0e4d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:29:37.683640 11871 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff8e9293c78424391997ee3eb0e4d95 (DEFAULT_TABLE table=test-table [id=807997729d644cdf947a29da15b3f09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:29:37.684525 12014 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff8e9293c78424391997ee3eb0e4d95 (DEFAULT_TABLE table=test-table [id=807997729d644cdf947a29da15b3f09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:29:37.685067 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff8e9293c78424391997ee3eb0e4d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:29:37.685792 12014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff8e9293c78424391997ee3eb0e4d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:29:37.694801 12231 tablet_bootstrap.cc:492] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Bootstrap starting.
I20260619 06:29:37.698405 12231 tablet_bootstrap.cc:654] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Neither blocks nor log segments found. Creating new log.
I20260619 06:29:37.699143 12232 tablet_bootstrap.cc:492] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Bootstrap starting.
I20260619 06:29:37.700273 12231 log.cc:824] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Log is configured to *not* fsync() on all Append() calls
I20260619 06:29:37.703478 12231 tablet_bootstrap.cc:492] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: No bootstrap required, opened a new log
I20260619 06:29:37.703480 12232 tablet_bootstrap.cc:654] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Neither blocks nor log segments found. Creating new log.
I20260619 06:29:37.703754 12231 ts_tablet_manager.cc:1397] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 06:29:37.704712 12232 log.cc:824] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:29:37.705168 12233 tablet_bootstrap.cc:492] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Bootstrap starting.
I20260619 06:29:37.707823 12232 tablet_bootstrap.cc:492] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: No bootstrap required, opened a new log
I20260619 06:29:37.708096 12233 tablet_bootstrap.cc:654] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Neither blocks nor log segments found. Creating new log.
I20260619 06:29:37.708142 12232 ts_tablet_manager.cc:1397] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260619 06:29:37.709192 12233 log.cc:824] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Log is configured to *not* fsync() on all Append() calls
I20260619 06:29:37.711416 12233 tablet_bootstrap.cc:492] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: No bootstrap required, opened a new log
I20260619 06:29:37.711650 12233 ts_tablet_manager.cc:1397] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260619 06:29:37.714634 12231 raft_consensus.cc:348] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.715067 12231 raft_consensus.cc:374] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:29:37.715195 12231 raft_consensus.cc:729] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47d50d09db64cc0aeef214a0a51e443, State: Initialized, Role: FOLLOWER
I20260619 06:29:37.715801 12231 consensus_queue.cc:260] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.716920 12213 heartbeater.cc:503] Master 127.0.97.126:42583 was elected leader, sending a full tablet report...
I20260619 06:29:37.719007 12231 ts_tablet_manager.cc:1428] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260619 06:29:37.719439 12232 raft_consensus.cc:348] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.719828 12232 raft_consensus.cc:374] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:29:37.719954 12232 raft_consensus.cc:729] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931d9ec3f51943449ccd90d51a37fe0e, State: Initialized, Role: FOLLOWER
I20260619 06:29:37.719964 12233 raft_consensus.cc:348] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.720352 12233 raft_consensus.cc:374] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:29:37.720503 12233 raft_consensus.cc:729] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225183533ca2413da4335e8ab8d0d318, State: Initialized, Role: FOLLOWER
I20260619 06:29:37.720587 12232 consensus_queue.cc:260] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.721288 12233 consensus_queue.cc:260] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.722514 12232 ts_tablet_manager.cc:1428] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 06:29:37.723483 12079 heartbeater.cc:503] Master 127.0.97.126:42583 was elected leader, sending a full tablet report...
I20260619 06:29:37.724109 12233 ts_tablet_manager.cc:1428] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
W20260619 06:29:37.816488 12080 tablet.cc:2369] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:29:37.833724 12214 tablet.cc:2369] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:29:37.835736 11937 tablet.cc:2369] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:29:37.838472 12237 raft_consensus.cc:482] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:29:37.838783 12237 raft_consensus.cc:504] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.840467 12237 leader_election.cc:288] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 225183533ca2413da4335e8ab8d0d318 (127.0.97.66:36763), 931d9ec3f51943449ccd90d51a37fe0e (127.0.97.65:34713)
I20260619 06:29:37.846705 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.840863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:53417 (local address 127.0.97.66:36763)
0619 06:29:37.841095 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.841101 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.841380 (+ 279us) server_negotiation.cc:407] Connection header received
0619 06:29:37.841639 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.841645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.841736 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.841825 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:29:37.842275 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.843124 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.844905 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.845800 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.845893 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.845979 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.846335 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.846453 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:29:37.846705 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.840948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:36763 (local address 127.0.97.67:53417)
0619 06:29:37.841233 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:37.841268 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:37.841420 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:37.841883 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:37.841889 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:37.841922 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:29:37.842116 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.842138 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.843238 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.843244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.844770 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.844780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.845934 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.845958 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.846163 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.846219 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 06:29:37.846314 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:29:37.846445 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":50}
I20260619 06:29:37.846925 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.840980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35355 (local address 127.0.97.65:34713)
0619 06:29:37.841201 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.841209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.841830 (+ 621us) server_negotiation.cc:407] Connection header received
0619 06:29:37.841948 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.841954 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.842030 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.842125 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:29:37.842690 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.843510 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.845197 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.846010 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.846130 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.846219 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.846559 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.846685 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:29:37.847594 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.840967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34713 (local address 127.0.97.67:35355)
0619 06:29:37.841701 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 06:29:37.841731 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:29:37.841845 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:29:37.842155 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:29:37.842158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:29:37.842174 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:29:37.842537 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.842547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.843639 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.843643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.845049 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:29:37.845061 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.846218 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.846224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:29:37.846369 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.846399 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:29:37.847088 (+ 689us) client_negotiation.cc:240] Negotiation successful
0619 06:29:37.847230 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":2,"thread_start_us":424,"threads_started":1}
I20260619 06:29:37.848033 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff8e9293c78424391997ee3eb0e4d95" candidate_uuid: "d47d50d09db64cc0aeef214a0a51e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225183533ca2413da4335e8ab8d0d318" is_pre_election: true
I20260619 06:29:37.849061 12034 raft_consensus.cc:2454] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d47d50d09db64cc0aeef214a0a51e443 in term 0.
I20260619 06:29:37.849145 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff8e9293c78424391997ee3eb0e4d95" candidate_uuid: "d47d50d09db64cc0aeef214a0a51e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931d9ec3f51943449ccd90d51a37fe0e" is_pre_election: true
I20260619 06:29:37.849640 11891 raft_consensus.cc:2454] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d47d50d09db64cc0aeef214a0a51e443 in term 0.
I20260619 06:29:37.849888 12104 leader_election.cc:302] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 225183533ca2413da4335e8ab8d0d318, d47d50d09db64cc0aeef214a0a51e443; no voters:
I20260619 06:29:37.850301 12237 raft_consensus.cc:2781] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:29:37.850431 12237 raft_consensus.cc:482] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:29:37.850534 12237 raft_consensus.cc:3037] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:29:37.852589 12237 raft_consensus.cc:504] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.853205 12237 leader_election.cc:288] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [CANDIDATE]: Term 1 election: Requested vote from peers 225183533ca2413da4335e8ab8d0d318 (127.0.97.66:36763), 931d9ec3f51943449ccd90d51a37fe0e (127.0.97.65:34713)
I20260619 06:29:37.853552 12034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff8e9293c78424391997ee3eb0e4d95" candidate_uuid: "d47d50d09db64cc0aeef214a0a51e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "225183533ca2413da4335e8ab8d0d318"
I20260619 06:29:37.853592 11891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff8e9293c78424391997ee3eb0e4d95" candidate_uuid: "d47d50d09db64cc0aeef214a0a51e443" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931d9ec3f51943449ccd90d51a37fe0e"
I20260619 06:29:37.853754 11891 raft_consensus.cc:3037] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:29:37.853765 12034 raft_consensus.cc:3037] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:29:37.856850 12034 raft_consensus.cc:2454] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d47d50d09db64cc0aeef214a0a51e443 in term 1.
I20260619 06:29:37.856850 11891 raft_consensus.cc:2454] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d47d50d09db64cc0aeef214a0a51e443 in term 1.
I20260619 06:29:37.857244 12104 leader_election.cc:302] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 225183533ca2413da4335e8ab8d0d318, 931d9ec3f51943449ccd90d51a37fe0e, d47d50d09db64cc0aeef214a0a51e443; no voters:
I20260619 06:29:37.857533 12237 raft_consensus.cc:2781] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:29:37.857978 12237 raft_consensus.cc:686] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 1 LEADER]: Becoming Leader. State: Replica: d47d50d09db64cc0aeef214a0a51e443, State: Running, Role: LEADER
I20260619 06:29:37.858392 12237 consensus_queue.cc:237] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } }
I20260619 06:29:37.862484 11722 catalog_manager.cc:5495] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 reported cstate change: term changed from 0 to 1, leader changed from <none> to d47d50d09db64cc0aeef214a0a51e443 (127.0.97.67). New cstate: current_term: 1 leader_uuid: "d47d50d09db64cc0aeef214a0a51e443" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d47d50d09db64cc0aeef214a0a51e443" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 } health_report { overall_health: UNKNOWN } } }
I20260619 06:29:37.904322 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.899105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.0.97.67:38199)
0619 06:29:37.899283 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.899289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.899375 (+ 86us) server_negotiation.cc:407] Connection header received
0619 06:29:37.899632 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.899635 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.899714 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.899855 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:29:37.900213 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.900885 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.902299 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.902675 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.902798 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.903326 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:29:37.903796 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:29:37.903946 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.903998 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.904068 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:29:37.911679 11891 raft_consensus.cc:1261] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 1 FOLLOWER]: Refusing update from remote peer d47d50d09db64cc0aeef214a0a51e443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:29:37.911702 12034 raft_consensus.cc:1261] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 1 FOLLOWER]: Refusing update from remote peer d47d50d09db64cc0aeef214a0a51e443: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:29:37.912535 12241 consensus_queue.cc:1035] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "931d9ec3f51943449ccd90d51a37fe0e" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:29:37.912883 12237 consensus_queue.cc:1035] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [LEADER]: Connected to new peer: Peer: permanent_uuid: "225183533ca2413da4335e8ab8d0d318" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 36763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:29:37.942107 11723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:29:37.942340 11723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:29:37.945758 11723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 62e2cda45ece4f588ead791114604341: Sending DeleteTablet for 3 replicas of tablet 5ff8e9293c78424391997ee3eb0e4d95
I20260619 06:29:37.946933 12148 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ff8e9293c78424391997ee3eb0e4d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:29:37 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38008
I20260619 06:29:37.947026 11871 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ff8e9293c78424391997ee3eb0e4d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:29:37 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:50474
I20260619 06:29:37.947813 12253 tablet_replica.cc:330] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: stopping tablet replica
I20260619 06:29:37.948195 12014 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ff8e9293c78424391997ee3eb0e4d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:29:37 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:56776
I20260619 06:29:37.948681 12253 raft_consensus.cc:2229] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:29:37.949324 12253 raft_consensus.cc:2258] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:29:37.950222 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 11803
I20260619 06:29:37.951596 12253 ts_tablet_manager.cc:1905] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:29:37.959889 12253 ts_tablet_manager.cc:1918] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:29:37.960119 12253 log.cc:1196] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/5ff8e9293c78424391997ee3eb0e4d95
I20260619 06:29:37.960791 12253 ts_tablet_manager.cc:1939] T 5ff8e9293c78424391997ee3eb0e4d95 P d47d50d09db64cc0aeef214a0a51e443: Deleting consensus metadata
I20260619 06:29:37.962569 11710 catalog_manager.cc:4841] TS d47d50d09db64cc0aeef214a0a51e443 (127.0.97.67:38199): tablet 5ff8e9293c78424391997ee3eb0e4d95 (table test-table [id=807997729d644cdf947a29da15b3f09d]) successfully deleted
I20260619 06:29:37.963990 12254 tablet_replica.cc:330] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: stopping tablet replica
I20260619 06:29:37.964437 12254 raft_consensus.cc:2229] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:29:37.964941 12254 raft_consensus.cc:2258] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:29:37.967063 12254 ts_tablet_manager.cc:1905] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:29:37.967350 12255 tablet_replica.cc:330] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: stopping tablet replica
I20260619 06:29:37.967839 12255 raft_consensus.cc:2229] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 1 FOLLOWER]: Raft consensus shutting down.
Jun 19 06:29:37 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:29:37.968357 12255 raft_consensus.cc:2258] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:29:37.970455 12255 ts_tablet_manager.cc:1905] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:29:37.974298 12254 ts_tablet_manager.cc:1918] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:29:37.974538 12254 log.cc:1196] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/5ff8e9293c78424391997ee3eb0e4d95
I20260619 06:29:37.975135 12254 ts_tablet_manager.cc:1939] T 5ff8e9293c78424391997ee3eb0e4d95 P 931d9ec3f51943449ccd90d51a37fe0e: Deleting consensus metadata
I20260619 06:29:37.976953 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:37.952922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.0.97.65:34713)
0619 06:29:37.953147 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:37.953156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:37.956553 (+ 3397us) server_negotiation.cc:407] Connection header received
0619 06:29:37.956750 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:37.956758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:37.956897 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:37.957007 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:37.959218 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.960695 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.963254 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:37.964078 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:37.964173 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:37.969524 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:37.969537 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:37.969552 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:37.969604 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:37.973454 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.974325 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.974336 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.974351 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.974503 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:37.974994 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:37.975002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:37.975006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:37.975472 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:37.975757 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:37.976341 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 06:29:37.976539 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 06:29:37.976931 11707 catalog_manager.cc:4841] TS 931d9ec3f51943449ccd90d51a37fe0e (127.0.97.65:34713): tablet 5ff8e9293c78424391997ee3eb0e4d95 (table test-table [id=807997729d644cdf947a29da15b3f09d]) successfully deleted
I20260619 06:29:37.977105 12255 ts_tablet_manager.cc:1918] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:29:37.977244 12255 log.cc:1196] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/5ff8e9293c78424391997ee3eb0e4d95
I20260619 06:29:37.977739 12255 ts_tablet_manager.cc:1939] T 5ff8e9293c78424391997ee3eb0e4d95 P 225183533ca2413da4335e8ab8d0d318: Deleting consensus metadata
I20260619 06:29:37.978282 11931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:29:37.979398 11710 catalog_manager.cc:4841] TS 225183533ca2413da4335e8ab8d0d318 (127.0.97.66:36763): tablet 5ff8e9293c78424391997ee3eb0e4d95 (table test-table [id=807997729d644cdf947a29da15b3f09d]) successfully deleted
I20260619 06:29:38.074313 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 11803
I20260619 06:29:38.098505 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 11942
Jun 19 06:29:38 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:29:38.116014 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:38.099320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.97.66:36763)
0619 06:29:38.099492 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:38.099496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:38.099514 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:29:38.099712 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:38.099716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:38.099797 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:38.099875 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:38.101535 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:38.102233 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:38.104240 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:38.104608 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:38.104705 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:38.109982 (+ 5277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:38.109997 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:38.110013 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:38.110049 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:38.113359 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:38.114149 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:38.114156 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:38.114158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:38.114316 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:38.114685 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:38.114688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:38.114690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:38.115018 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:38.115233 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:38.115601 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 06:29:38.115759 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:29:38.116822 12074 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:29:38.206254 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 11942
I20260619 06:29:38.228449 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 12085
Jun 19 06:29:38 dist-test-slave-4t1f krb5kdc[11247](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:29:38.246625 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:29:38.229271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.0.97.67:38199)
0619 06:29:38.229438 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:29:38.229443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:29:38.229551 (+ 108us) server_negotiation.cc:407] Connection header received
0619 06:29:38.229764 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:29:38.229769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:29:38.229850 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:29:38.229956 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:29:38.231847 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:38.232688 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:38.234742 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:29:38.235296 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:29:38.235382 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:29:38.240846 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:29:38.240856 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:29:38.240869 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:29:38.240902 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:29:38.243915 (+ 3013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:38.244574 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:38.244589 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:38.244599 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:38.244743 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:29:38.245144 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:29:38.245147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:29:38.245149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:29:38.245477 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:29:38.245784 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:29:38.246059 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:29:38.246271 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:29:38.247308 12208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:29:38.362291 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 12085
I20260619 06:29:38.380537 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 11691
I20260619 06:29:38.381644 11753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:29:38.490805 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 11691
I20260619 06:29:38.512260 389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:29:38.847513 389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:29:38.847711 389 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:29:39.192977 389 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:29:39.193 UTC [11277] LOG: received smart shutdown request
2026-06-19 06:29:39.196 UTC [11277] LOG: background worker "logical replication launcher" (PID 11286) exited with exit code 1
2026-06-19 06:29:39.196 UTC [11281] LOG: shutting down
2026-06-19 06:29:39.341 UTC [11277] LOG: database system is shut down
2026-06-19T06:29:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48106 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:29:39 dist-test-slave-4t1f krb5kdc[12269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:29:39 dist-test-slave-4t1f krb5kdc[12269](info): set up 2 sockets
Jun 19 06:29:39 dist-test-slave-4t1f krb5kdc[12269](info): commencing operation
krb5kdc: starting...
W20260619 06:29:40.553356 389 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:29:40 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:29:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:29:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/rangeradmin_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/rangerlookup_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
I20260619 06:29:40.851925 389 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskWesBqs/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/postgres -l logfile start
2026-06-19 06:29:43.185 UTC [12299] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 06:29:43.186 UTC [12299] LOG: listening on IPv4 address "127.0.97.84", port 58003
2026-06-19 06:29:43.189 UTC [12299] LOG: listening on Unix socket "/tmp/.s.PGSQL.58003"
2026-06-19 06:29:43.209 UTC [12302] LOG: database system was shut down at 2026-06-19 06:29:41 UTC
2026-06-19 06:29:43.212 UTC [12299] LOG: database system is ready to accept connections
I20260619 06:29:44.196944 389 mini_postgres.cc:96] Postgres bound to 58003
2026-06-19 06:29:44.213 UTC [12310] FATAL: database "slave" does not exist
127.0.97.84:58003 - accepting connections
I20260619 06:29:44.214147 389 mini_ranger.cc:161] Starting Ranger...
I20260619 06:29:44.239224 389 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 06:29:44.663946 389 mini_ranger.cc:87] Created ranger Postgres database
I20260619 06:29:44.664058 389 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-admin
2026-06-19 06:29:44,951 [I] DB FLAVOR :POSTGRES
2026-06-19 06:29:44,952 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:29:44,952 [I] Checking connection..
2026-06-19 06:29:44,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 06:29:45,213 [I] Checking connection passed.
2026-06-19 06:29:45,213 [I] --------- Verifying version history table ---------
2026-06-19 06:29:45,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:29:45,467 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:29:45,467 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:29:45,705 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 06:29:45,705 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 06:29:45,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 06:29:45.964 UTC [12374] WARNING: there is no transaction in progress
2026-06-19 06:29:45,970 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 06:29:45,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 06:29:46,214 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 06:29:46,214 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:29:46,214 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 06:29:46,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 06:29:46,706 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4t1f.c.gcp-upstream.internal','N') ;"
2026-06-19 06:29:46,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:29:47,194 [I] Table x_portal_user does not exist in database ranger
2026-06-19 06:29:47,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:29:47,429 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 06:29:47,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:29:47,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:29:47,869 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 06:29:47,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 06:29:48.750 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.765 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.781 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.795 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.812 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.831 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.892 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.899 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.912 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.922 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.931 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.944 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.953 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.959 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.964 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:48.970 UTC [12482] WARNING: there is no transaction in progress
2026-06-19 06:29:49,560 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 06:29:49,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 06:29:49,789 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 06:29:49,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 06:29:50,048 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 06:29:50,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 06:29:50,291 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 06:29:50,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 06:29:50,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 06:29:50,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4t1f.c.gcp-upstream.internal';"
2026-06-19 06:29:51,017 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 06:29:51,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 06:29:51,241 [I] DB_PATCHES have already been applied
I20260619 06:29:51.246865 389 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-admin:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskWesBqs/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 06:29:51.246944 389 mini_ranger.cc:193] Using host: 127.0.97.84
I20260619 06:29:51.260970 389 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.84:34469
Jun 19, 2026 6:29:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:29:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:29:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 6:29:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 6:29:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 6:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 6:29:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 06:29:52.379127 389 mini_ranger.cc:160] Time spent starting Ranger: real 8.165s user 0.005s sys 0.031s
Jun 19, 2026 6:30:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 6:30:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 6:30:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 6:30:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 6:30:12 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 06:30:13.502605 389 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/rangerkms_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/HTTP_127.0.97.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 06:30:13 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850613, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 06:30:13.668345 389 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 06:30:13.690593 389 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 06:30:14.492219 389 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 06:30:14.492337 389 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms
I20260619 06:30:14.492383 389 mini_ranger_kms.cc:227] Using postgres at 127.0.97.84:58003
2026-06-19 06:30:14,883 [I] DB FLAVOR :POSTGRES
2026-06-19 06:30:14,883 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 06:30:14,883 [I] Checking connection
2026-06-19 06:30:14,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 06:30:15,130 [I] connection success
2026-06-19 06:30:15,131 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 06:30:15,131 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 06:30:15,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 06:30:15,369 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 06:30:15,369 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 06:30:15,369 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 06:30:15,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.84:58003/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 06:30:15,614 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 06:30:16.045773 389 mini_ranger_kms.cc:310] Created kms service
I20260619 06:30:16.254245 389 mini_ranger_kms.cc:326] Created kudu user
I20260619 06:30:16.299582 389 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 06:30:16 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850613, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:16.676959 389 mini_ranger_kms.cc:384] Added ranger policy
I20260619 06:30:16.677320 389 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms:/tmp/dist-test-taskWesBqs/build/asan/bin/postgresql.jar:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskWesBqs/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 06:30:16.677383 389 mini_ranger_kms.cc:240] Using host: 127.0.97.84
I20260619 06:30:16.693606 389 mini_ranger_kms.cc:276] Ranger KMS PID: 12662
I20260619 06:30:16.693723 389 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.84:45085
Jun 19, 2026 6:30:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 6:30:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 6:30:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 6:30:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 6:30:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 6:30:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260619 06:30:19.025367 389 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.357s user 0.073s sys 0.333s
I20260619 06:30:19.025614 389 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 06:30:19.025813 389 mini_ranger_kms.cc:401] 127.0.97.84:45085/kms/v1/keys
Jun 19 06:30:19 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 19 06:30:19 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 19 06:30:19 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850619, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.84@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 19 06:30:22 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:30:22.125510 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40169
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:40169
--ranger_config_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:22.449811 12713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:22.450135 12713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:22.450299 12713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:22.457605 12713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:22.457732 12713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:22.457849 12713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:22.457942 12713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:22.458024 12713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:22.468031 12713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.84:45085/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:40169
--ranger_config_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:22.469362 12713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:22.470719 12713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:22.482301 12721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:22.482376 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:22.482376 12719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:22.483428 12713 server_base.cc:1034] running on GCE node
I20260619 06:30:22.484176 12713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:22.485635 12713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:22.486899 12713 hybrid_clock.cc:648] HybridClock initialized: now 1781850622486852 us; error 55 us; skew 500 ppm
I20260619 06:30:22.487313 12713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:22 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:22.490433 12713 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:30:22.491891 12713 webserver.cc:460] Webserver started at http://127.0.97.126:38017/ using document root <none> and password file <none>
I20260619 06:30:22.492478 12713 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:22.492568 12713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:22.492808 12713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:30:22 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850622, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:22.548210 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "500e55da023b4d3487f4360b663f2ba8"
format_stamp: "Formatted at 2026-06-19 06:30:22 on dist-test-slave-4t1f"
server_key: "3b0e82a55480af75628ef73e222d522a"
server_key_iv: "37467a4b02c2cee21e37d334bec02c96"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:22.548995 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "500e55da023b4d3487f4360b663f2ba8"
format_stamp: "Formatted at 2026-06-19 06:30:22 on dist-test-slave-4t1f"
server_key: "3b0e82a55480af75628ef73e222d522a"
server_key_iv: "37467a4b02c2cee21e37d334bec02c96"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:22.554313 12713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 06:30:22.574856 12728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:22.576468 12713 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:30:22.576670 12713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "500e55da023b4d3487f4360b663f2ba8"
format_stamp: "Formatted at 2026-06-19 06:30:22 on dist-test-slave-4t1f"
server_key: "3b0e82a55480af75628ef73e222d522a"
server_key_iv: "37467a4b02c2cee21e37d334bec02c96"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:22.576893 12713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:22.593472 12713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:22.596892 12713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:22.597247 12713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:22.615170 12713 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:40169
I20260619 06:30:22.615192 12779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:40169 every 8 connection(s)
I20260619 06:30:22.616487 12713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:30:22.619897 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 12713
I20260619 06:30:22.619977 12780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:22.620133 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:30:22.620796 389 external_mini_cluster.cc:1428] Setting key 1124a88f7eaa855f48a4dd1408077800
I20260619 06:30:22.632980 12780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Bootstrap starting.
Jun 19 06:30:22 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:22.636379 12780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:22.637488 12780 log.cc:824] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:22.640480 12780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: No bootstrap required, opened a new log
I20260619 06:30:22.644003 12783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:22.622713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.0.97.126:40169)
0619 06:30:22.623477 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:22.623488 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:22.623526 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:30:22.625660 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:22.625681 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:22.626064 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:22.626380 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:22.627903 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:22.629183 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:22.631120 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:22.631758 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:22.631885 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:22.636201 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:22.636208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:22.636221 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:22.636253 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:22.639780 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:22.640422 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:22.640432 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:22.640443 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:22.640639 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:22.640982 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:22.640989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:22.640993 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:22.641638 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:22.641940 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:22.642287 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 06:30:22.642692 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":370,"threads_started":1}
I20260619 06:30:22.648808 12780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:22.649251 12780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:22.649330 12780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500e55da023b4d3487f4360b663f2ba8, State: Initialized, Role: FOLLOWER
I20260619 06:30:22.649998 12780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:22.650219 12780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:30:22.650313 12780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:30:22.650434 12780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:22.652508 12780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:22.652922 12780 leader_election.cc:302] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500e55da023b4d3487f4360b663f2ba8; no voters:
I20260619 06:30:22.653457 12780 leader_election.cc:288] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:30:22.653576 12785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:30:22.654245 12785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 500e55da023b4d3487f4360b663f2ba8, State: Running, Role: LEADER
I20260619 06:30:22.654724 12785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:22.655401 12780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:22.657759 12787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500e55da023b4d3487f4360b663f2ba8. Latest consensus state: current_term: 1 leader_uuid: "500e55da023b4d3487f4360b663f2ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } } }
I20260619 06:30:22.658182 12787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:22.658988 12786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500e55da023b4d3487f4360b663f2ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } } }
I20260619 06:30:22.659318 12786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:22.660760 12792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:30:22.665200 12792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:30:22.667053 12780 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 06:30:22.682116 12792 catalog_manager.cc:1347] Generated new cluster ID: 804c022ecd504a5e84b012466797a7d1
I20260619 06:30:22.682252 12792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:30:22.707074 12792 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:30:22.709408 12792 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:30:22.725531 12792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Generated new TSK 0
I20260619 06:30:22.726725 12792 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:30:23.452116 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40169
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:30:23 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 06:30:23.848488 12825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:23.848950 12825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:23.849169 12825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:23.858328 12825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:23.858461 12825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:23.858649 12825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:30:23.873347 12825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.84:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:23.874996 12825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:23.876704 12825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:23.902145 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:23.903790 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:23.911949 12825 server_base.cc:1034] running on GCE node
W20260619 06:30:23.912523 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:23.913401 12825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:23.914711 12825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:23.916026 12825 hybrid_clock.cc:648] HybridClock initialized: now 1781850623915916 us; error 94 us; skew 500 ppm
I20260619 06:30:23.916424 12825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:23 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:23.920686 12825 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:30:23.930610 12825 webserver.cc:460] Webserver started at http://127.0.97.65:43349/ using document root <none> and password file <none>
I20260619 06:30:23.931515 12825 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:23.931679 12825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:23.932039 12825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:30:23 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:23.954514 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1"
format_stamp: "Formatted at 2026-06-19 06:30:23 on dist-test-slave-4t1f"
server_key: "b82c34dfd3fe3e2a5ff47e3217031764"
server_key_iv: "ecdaac35e7488e6c1da76cf94958634a"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:23.956568 12825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1"
format_stamp: "Formatted at 2026-06-19 06:30:23 on dist-test-slave-4t1f"
server_key: "b82c34dfd3fe3e2a5ff47e3217031764"
server_key_iv: "ecdaac35e7488e6c1da76cf94958634a"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:23.964226 12825 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.005s
I20260619 06:30:23.989787 12845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:23.992398 12825 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.006s sys 0.000s
I20260619 06:30:23.993733 12825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1"
format_stamp: "Formatted at 2026-06-19 06:30:23 on dist-test-slave-4t1f"
server_key: "b82c34dfd3fe3e2a5ff47e3217031764"
server_key_iv: "ecdaac35e7488e6c1da76cf94958634a"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:23.994068 12825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:24.048878 12825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:24.054312 12825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:24.054731 12825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:24.060379 12825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:24.062253 12825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:24.062361 12825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:24.062526 12825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:24.062593 12825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:24.117928 12825 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:43627
I20260619 06:30:24.117950 12957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:43627 every 8 connection(s)
I20260619 06:30:24.119709 12825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:30:24.124162 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 12825
I20260619 06:30:24.124374 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:30:24.125173 389 external_mini_cluster.cc:1428] Setting key 92061ef5f9d4140075de54183d293d4e
Jun 19 06:30:24 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:24.158952 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:24.123457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40169 (local address 127.0.97.65:48731)
0619 06:30:24.134954 (+ 11497us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:24.135016 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:24.137446 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:24.138559 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:24.138573 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:24.139213 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:24.140082 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:24.140106 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:24.141736 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:24.141743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:24.143882 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:24.143927 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:24.145289 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:24.145297 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:24.145586 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:24.146395 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:24.146425 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:24.149700 (+ 3275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:24.153932 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:24.153944 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:24.153964 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:24.154582 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:24.155033 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:24.155040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:24.155047 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:24.155306 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:24.156074 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:24.156084 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:24.156575 (+ 491us) client_negotiation.cc:769] Sending connection context
0619 06:30:24.156906 (+ 331us) client_negotiation.cc:240] Negotiation successful
0619 06:30:24.157514 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11177,"thread_start_us":13653,"threads_started":1}
I20260619 06:30:24.160107 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:24.123596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:48731 (local address 127.0.97.126:40169)
0619 06:30:24.134360 (+ 10764us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:24.134367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:24.137665 (+ 3298us) server_negotiation.cc:407] Connection header received
0619 06:30:24.137918 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:24.137924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:24.138015 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:24.138121 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:24.140302 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:24.141528 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:24.144083 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:24.144855 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:24.144973 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:24.149966 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:24.149974 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:24.149979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:24.150031 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:24.153739 (+ 3708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:24.154732 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:24.154737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:24.154740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:24.154894 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:24.155443 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:24.155451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:24.155457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:24.155864 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:24.156007 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:24.159524 (+ 3517us) server_negotiation.cc:299] Negotiation successful
0619 06:30:24.159748 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10616,"thread_start_us":10521,"threads_started":1}
I20260619 06:30:24.162468 12958 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40169
I20260619 06:30:24.162773 12958 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:24.163633 12958 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
I20260619 06:30:24.166317 12745 ts_manager.cc:194] Registered new tserver with Master: e4c21bfad0974e0cb6e13bd20f7c83e1 (127.0.97.65:43627)
I20260619 06:30:24.168509 12745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:48731
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:30:24.213835 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40169
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:30:24 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
Jun 19 06:30:24 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260619 06:30:24.715587 12964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:24.716135 12964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:24.716333 12964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:24.727944 12964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:24.728082 12964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:24.728320 12964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:30:24.743932 12964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.84:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:24.745970 12964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:24.747740 12964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:24.781992 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:24.783195 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:24.793327 12964 server_base.cc:1034] running on GCE node
W20260619 06:30:24.794396 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:24.795281 12964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:24.804152 12964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:24.805481 12964 hybrid_clock.cc:648] HybridClock initialized: now 1781850624805399 us; error 72 us; skew 500 ppm
I20260619 06:30:24.805928 12964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:24 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:24.814458 12964 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:30:24.829499 12964 webserver.cc:460] Webserver started at http://127.0.97.66:44799/ using document root <none> and password file <none>
I20260619 06:30:24.830627 12964 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:24.830859 12964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:24.831300 12964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:30:24 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:24.850183 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "8e553d42833f4f3d92a937a3af295459"
format_stamp: "Formatted at 2026-06-19 06:30:24 on dist-test-slave-4t1f"
server_key: "188187d1b3316040fed99578f52ef1f7"
server_key_iv: "9634aa7278bf72ba9290b443e17be4b2"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:24.851406 12964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "8e553d42833f4f3d92a937a3af295459"
format_stamp: "Formatted at 2026-06-19 06:30:24 on dist-test-slave-4t1f"
server_key: "188187d1b3316040fed99578f52ef1f7"
server_key_iv: "9634aa7278bf72ba9290b443e17be4b2"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:24.866075 12964 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.004s
I20260619 06:30:24.890007 12988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:24.892059 12964 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.002s sys 0.006s
I20260619 06:30:24.892267 12964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "8e553d42833f4f3d92a937a3af295459"
format_stamp: "Formatted at 2026-06-19 06:30:24 on dist-test-slave-4t1f"
server_key: "188187d1b3316040fed99578f52ef1f7"
server_key_iv: "9634aa7278bf72ba9290b443e17be4b2"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:24.892558 12964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:24.928493 12964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:24.933730 12964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:24.934060 12964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:24.935241 12964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:24.937964 12964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:24.938057 12964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:24.938241 12964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:24.938335 12964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:24.999056 12964 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44639
I20260619 06:30:24.999115 13100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44639 every 8 connection(s)
I20260619 06:30:25.000763 12964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:30:25.010260 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 12964
I20260619 06:30:25.010433 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:30:25.010928 389 external_mini_cluster.cc:1428] Setting key 32abadfb991b4a6ad4f3bf52df04dbdd
Jun 19 06:30:25 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
I20260619 06:30:25.044713 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:25.004523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40169 (local address 127.0.97.66:51841)
0619 06:30:25.019765 (+ 15242us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:25.019829 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:25.022458 (+ 2629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:25.023613 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:25.023625 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:25.024269 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:25.025127 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:25.025154 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.026663 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.026671 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:25.028756 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:25.028770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.029925 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.029932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:25.030187 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:25.030999 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:25.031033 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:25.034056 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:25.040218 (+ 6162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:25.040225 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:25.040239 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:25.040625 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:25.041101 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:25.041106 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:25.041110 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:25.041321 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:25.041929 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:25.041937 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:25.042324 (+ 387us) client_negotiation.cc:769] Sending connection context
0619 06:30:25.042637 (+ 313us) client_negotiation.cc:240] Negotiation successful
0619 06:30:25.043182 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14895,"thread_start_us":16955,"threads_started":1}
I20260619 06:30:25.046749 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:25.003852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51841 (local address 127.0.97.126:40169)
0619 06:30:25.019106 (+ 15254us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:25.019113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:25.022578 (+ 3465us) server_negotiation.cc:407] Connection header received
0619 06:30:25.022945 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:25.022951 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:25.023070 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:25.023176 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:25.025317 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.026469 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.028919 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.029540 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.029653 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:25.034319 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:25.034326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:25.034331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:25.034369 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:25.038762 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.040779 (+ 2017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.040784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.040787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.040958 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.041445 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.041450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.041453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.041795 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:25.041956 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:25.045978 (+ 4022us) server_negotiation.cc:299] Negotiation successful
0619 06:30:25.046254 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15123,"thread_start_us":15240,"threads_started":1}
I20260619 06:30:25.048277 13101 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40169
I20260619 06:30:25.048594 13101 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:25.049443 13101 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
I20260619 06:30:25.051096 12745 ts_manager.cc:194] Registered new tserver with Master: 8e553d42833f4f3d92a937a3af295459 (127.0.97.66:44639)
I20260619 06:30:25.052287 12745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:51841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:30:25.072461 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40169
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:30:25.172649 12958 heartbeater.cc:503] Master 127.0.97.126:40169 was elected leader, sending a full tablet report...
W20260619 06:30:25.412528 13107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:25.412822 13107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:25.413004 13107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:25.420466 13107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:25.420629 13107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:25.420817 13107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:30:25.430810 13107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.84:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:25.432150 13107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:25.433506 13107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:25.446632 13113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:25.446659 13112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:25.446714 13115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:25.447567 13107 server_base.cc:1034] running on GCE node
I20260619 06:30:25.448182 13107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:25.449169 13107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:25.450429 13107 hybrid_clock.cc:648] HybridClock initialized: now 1781850625450360 us; error 64 us; skew 500 ppm
I20260619 06:30:25.450731 13107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:25 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:25.453948 13107 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:30:25.455323 13107 webserver.cc:460] Webserver started at http://127.0.97.67:44081/ using document root <none> and password file <none>
I20260619 06:30:25.456085 13107 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:25.456249 13107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:25.456573 13107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 06:30:25 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:25.471237 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "6815f4e1697b419bab93e6be1a06434e"
format_stamp: "Formatted at 2026-06-19 06:30:25 on dist-test-slave-4t1f"
server_key: "82884b059c12889f61458b7be1a092f2"
server_key_iv: "8f747385c750a7c1b7a24f708a96a5a8"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:25.472231 13107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "6815f4e1697b419bab93e6be1a06434e"
format_stamp: "Formatted at 2026-06-19 06:30:25 on dist-test-slave-4t1f"
server_key: "82884b059c12889f61458b7be1a092f2"
server_key_iv: "8f747385c750a7c1b7a24f708a96a5a8"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:25.477892 13107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:30:25.494642 13122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:25.496228 13107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:25.496408 13107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "6815f4e1697b419bab93e6be1a06434e"
format_stamp: "Formatted at 2026-06-19 06:30:25 on dist-test-slave-4t1f"
server_key: "82884b059c12889f61458b7be1a092f2"
server_key_iv: "8f747385c750a7c1b7a24f708a96a5a8"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:25.496636 13107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:25.516358 13107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:25.519742 13107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:25.520025 13107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:25.520912 13107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:25.522193 13107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:25.522264 13107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:25.522401 13107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:25.522455 13107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:25.554279 13107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44551
I20260619 06:30:25.554319 13234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44551 every 8 connection(s)
I20260619 06:30:25.555503 13107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:30:25.556880 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13107
I20260619 06:30:25.557051 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:30:25.557595 389 external_mini_cluster.cc:1428] Setting key a8a2612fb638a2b54b6fa151cb8ab8d8
Jun 19 06:30:25 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850625, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:25.575489 13237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:25.558162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:37415 (local address 127.0.97.126:40169)
0619 06:30:25.558689 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:25.558697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:25.561123 (+ 2426us) server_negotiation.cc:407] Connection header received
0619 06:30:25.562017 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:25.562024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:25.562116 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:25.562238 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:25.563573 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.564492 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.566066 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.566479 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.566610 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:25.569879 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:25.569883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:25.569886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:25.569915 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:25.572912 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.573643 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.573647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.573649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.573766 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.574144 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.574146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.574148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.574410 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:25.574514 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:25.575071 (+ 557us) server_negotiation.cc:299] Negotiation successful
0619 06:30:25.575223 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":273,"threads_started":1}
I20260619 06:30:25.576344 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:25.558803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40169 (local address 127.0.97.67:37415)
0619 06:30:25.559841 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:25.559901 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:25.561765 (+ 1864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:25.562468 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:25.562477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:25.562862 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:25.563398 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:25.563429 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.564665 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.564669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:25.565915 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:25.565926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.566692 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.566699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:25.566923 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:25.567621 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:25.567644 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:25.569700 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:25.573039 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:25.573047 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:25.573087 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:25.573504 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:25.573865 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:25.573868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:25.573870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:25.574046 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:25.574511 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:25.574516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:25.574785 (+ 269us) client_negotiation.cc:769] Sending connection context
0619 06:30:25.574990 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 06:30:25.575307 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":327,"threads_started":1}
I20260619 06:30:25.577601 13235 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40169
I20260619 06:30:25.577880 13235 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:25.578526 13235 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
I20260619 06:30:25.579838 12745 ts_manager.cc:194] Registered new tserver with Master: 6815f4e1697b419bab93e6be1a06434e (127.0.97.67:44551)
I20260619 06:30:25.580684 12745 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:37415
I20260619 06:30:25.584918 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:30:25.586508 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 12825
Jun 19 06:30:25 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:30:25.603637 13240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:25.587315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.0.97.65:43627)
0619 06:30:25.587897 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:25.587904 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:25.587927 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:30:25.587979 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:25.587985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:25.588162 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:25.588281 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:25.589751 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.591685 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.593217 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.593734 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.593830 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:25.597815 (+ 3985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:25.597825 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:25.597838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:25.597873 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:25.601221 (+ 3348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.601937 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.601943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.601945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.602079 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.602384 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.602387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.602389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.602683 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:25.602897 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:25.603160 (+ 263us) server_negotiation.cc:299] Negotiation successful
0619 06:30:25.603348 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":297,"threads_started":1}
I20260619 06:30:25.604831 12953 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:25.688177 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 12825
I20260619 06:30:25.704054 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 12964
Jun 19 06:30:25 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:30:25.720168 13242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:25.705006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41832 (local address 127.0.97.66:44639)
0619 06:30:25.705638 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:25.705645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:25.705670 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:30:25.705742 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:25.705748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:25.705959 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:25.706091 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:25.707398 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.708434 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.710011 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.710538 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.710636 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:25.714734 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:25.714745 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:25.714758 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:25.714788 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:25.717858 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.718476 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.718483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.718485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.718615 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.718958 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.718962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.718964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.719275 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:25.719489 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:25.719708 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 06:30:25.719888 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":383,"threads_started":1}
I20260619 06:30:25.721515 13096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:25.802084 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 12964
I20260619 06:30:25.821374 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13107
Jun 19 06:30:25 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:30:25.837112 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:25.822361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.0.97.67:44551)
0619 06:30:25.823008 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:25.823015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:25.823041 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:30:25.823099 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:25.823105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:25.823314 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:25.823452 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:25.824888 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.825931 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.827251 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:25.827773 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:25.827881 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:25.832138 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:25.832149 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:25.832163 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:25.832209 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:25.834836 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.835415 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.835422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.835424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.835552 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:25.835883 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:25.835886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:25.835888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:25.836178 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:25.836414 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:25.836614 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 06:30:25.836794 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":343,"threads_started":1}
I20260619 06:30:25.838335 13230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:25.920156 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13107
I20260619 06:30:25.935742 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 12713
I20260619 06:30:25.936965 12775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:26.041409 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 12713
I20260619 06:30:26.064666 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40169
--webserver_interface=127.0.97.126
--webserver_port=38017
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:40169
--ranger_config_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:26.382012 13247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:26.382357 13247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:26.382539 13247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:26.389595 13247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:26.389741 13247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:26.389849 13247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:26.389961 13247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:26.390054 13247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:26.399868 13247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:40169
--ranger_config_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:26.401155 13247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:26.402457 13247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:26.413740 13252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:26.413784 13253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:26.413821 13255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:26.414953 13247 server_base.cc:1034] running on GCE node
I20260619 06:30:26.415894 13247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:26.417286 13247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:26.418541 13247 hybrid_clock.cc:648] HybridClock initialized: now 1781850626418504 us; error 48 us; skew 500 ppm
I20260619 06:30:26.418926 13247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:26 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:26.422024 13247 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:30:26.423445 13247 webserver.cc:460] Webserver started at http://127.0.97.126:38017/ using document root <none> and password file <none>
I20260619 06:30:26.424074 13247 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:26.424222 13247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:30:26 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:26.507236 13247 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:30:26.510236 13263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:26.511595 13247 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:26.511787 13247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "500e55da023b4d3487f4360b663f2ba8"
format_stamp: "Formatted at 2026-06-19 06:30:22 on dist-test-slave-4t1f"
server_key: "3b0e82a55480af75628ef73e222d522a"
server_key_iv: "37467a4b02c2cee21e37d334bec02c96"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:26.512766 13247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:26.536859 13247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:26.540367 13247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:26.540647 13247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:26.558703 13247 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:40169
I20260619 06:30:26.558727 13314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:40169 every 8 connection(s)
I20260619 06:30:26.560084 13247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:30:26.566612 13315 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:30:26.568497 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13247
I20260619 06:30:26.569496 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:43627
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:40169
--webserver_port=43349
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:30:26.570416 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Bootstrap starting.
I20260619 06:30:26.590553 13315 log.cc:824] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:26.599043 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:30:26.599568 13315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Bootstrap complete.
I20260619 06:30:26.608587 13315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:26.609504 13315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500e55da023b4d3487f4360b663f2ba8, State: Initialized, Role: FOLLOWER
I20260619 06:30:26.610137 13315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:26.610347 13315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:30:26.610455 13315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:30:26.610625 13315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:30:26.614324 13315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:26.614778 13315 leader_election.cc:302] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500e55da023b4d3487f4360b663f2ba8; no voters:
I20260619 06:30:26.615378 13315 leader_election.cc:288] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:30:26.615547 13319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:30:26.616700 13319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [term 2 LEADER]: Becoming Leader. State: Replica: 500e55da023b4d3487f4360b663f2ba8, State: Running, Role: LEADER
I20260619 06:30:26.617301 13319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } }
I20260619 06:30:26.617779 13315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:26.620131 13320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "500e55da023b4d3487f4360b663f2ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } } }
I20260619 06:30:26.620126 13321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500e55da023b4d3487f4360b663f2ba8. Latest consensus state: current_term: 2 leader_uuid: "500e55da023b4d3487f4360b663f2ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500e55da023b4d3487f4360b663f2ba8" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40169 } } }
I20260619 06:30:26.620597 13320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:26.620667 13321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:26.621716 13325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:30:26.640442 13325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:30:26.642278 13325 catalog_manager.cc:1259] Loaded cluster ID: 804c022ecd504a5e84b012466797a7d1
I20260619 06:30:26.642452 13325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:30:26.650933 13325 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:30:26.652762 13325 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Loaded TSK: 0
I20260619 06:30:26.654699 13325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:30:26.658275 13315 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260619 06:30:27.011900 13317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:27.012341 13317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:27.012517 13317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:27.023993 13317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:27.024220 13317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:27.024426 13317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:30:27.040421 13317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:43627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=43349
--tserver_master_addrs=127.0.97.126:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:27.042074 13317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:27.043849 13317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:27.096956 13353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:27.102880 13352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:27.105381 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:27.105659 13317 server_base.cc:1034] running on GCE node
I20260619 06:30:27.106478 13317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:27.117743 13317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:27.121390 13317 hybrid_clock.cc:648] HybridClock initialized: now 1781850627121300 us; error 88 us; skew 500 ppm
I20260619 06:30:27.121891 13317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:27 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:27.126470 13317 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:30:27.128567 13317 webserver.cc:460] Webserver started at http://127.0.97.65:43349/ using document root <none> and password file <none>
I20260619 06:30:27.129570 13317 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:27.129739 13317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:30:27 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:27.188225 13317 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.007s sys 0.002s
I20260619 06:30:27.200125 13364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:27.202267 13317 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.005s
I20260619 06:30:27.202474 13317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1"
format_stamp: "Formatted at 2026-06-19 06:30:23 on dist-test-slave-4t1f"
server_key: "b82c34dfd3fe3e2a5ff47e3217031764"
server_key_iv: "ecdaac35e7488e6c1da76cf94958634a"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:27.204043 13317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:27.245220 13317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:27.250483 13317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:27.250784 13317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:27.258087 13317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:27.266101 13317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:27.266281 13317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:27.266404 13317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:27.266505 13317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:27.334765 13317 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:43627
I20260619 06:30:27.334811 13476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:43627 every 8 connection(s)
I20260619 06:30:27.336261 13317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:30:27.341984 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13317
I20260619 06:30:27.342983 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:44639
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:40169
--webserver_port=44799
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:30:27 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:27.369992 13479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:27.340340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40169 (local address 127.0.97.65:47863)
0619 06:30:27.341877 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:27.341927 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:27.344363 (+ 2436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:27.346044 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:27.346054 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:27.346932 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:27.347905 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:27.347934 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.350451 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.350456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:27.352458 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:27.352473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.354908 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.354917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:27.355236 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:27.356064 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:27.356099 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:27.359056 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:27.364064 (+ 5008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:27.364076 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:27.364093 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:27.364781 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:27.365410 (+ 629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:27.365415 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:27.365419 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:27.365707 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:27.366909 (+ 1202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:27.366920 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:27.367376 (+ 456us) client_negotiation.cc:769] Sending connection context
0619 06:30:27.367706 (+ 330us) client_negotiation.cc:240] Negotiation successful
0619 06:30:27.368328 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":2406,"threads_started":1}
I20260619 06:30:27.371134 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:27.341789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47863 (local address 127.0.97.126:40169)
0619 06:30:27.344507 (+ 2718us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:27.344525 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:27.344577 (+ 52us) server_negotiation.cc:407] Connection header received
0619 06:30:27.344984 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:27.344997 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:27.345475 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:27.345868 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:27.348184 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.350261 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.353218 (+ 2957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.354359 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.354537 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:27.359371 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:27.359383 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:27.359405 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:27.359462 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:27.363876 (+ 4414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:27.364986 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:27.364997 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:27.365006 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:27.365245 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:27.365869 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:27.365877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:27.365882 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:27.366585 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:27.366812 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:27.369447 (+ 2635us) server_negotiation.cc:299] Negotiation successful
0619 06:30:27.369860 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2451,"thread_start_us":2188,"threads_started":1}
I20260619 06:30:27.374610 13477 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40169
I20260619 06:30:27.374950 13477 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:27.375819 13477 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
W20260619 06:30:27.379114 13477 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:40169 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:30:27.379375 13477 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:27.379662 13477 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
I20260619 06:30:27.380877 13477 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:27.381209 13477 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
W20260619 06:30:27.382227 13477 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:30:27.705902 13481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:27.706193 13481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:27.706382 13481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:27.714780 13481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:27.714918 13481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:27.715143 13481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:30:27.725544 13481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:44639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=44799
--tserver_master_addrs=127.0.97.126:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:27.726823 13481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:27.728091 13481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:27.743155 13496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:27.743366 13495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:27.745432 13498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:27.746721 13481 server_base.cc:1034] running on GCE node
I20260619 06:30:27.747380 13481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:27.748521 13481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:27.749768 13481 hybrid_clock.cc:648] HybridClock initialized: now 1781850627749702 us; error 51 us; skew 500 ppm
I20260619 06:30:27.750125 13481 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:27 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:27.754307 13481 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:30:27.777449 13481 webserver.cc:460] Webserver started at http://127.0.97.66:44799/ using document root <none> and password file <none>
I20260619 06:30:27.778648 13481 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:27.778862 13481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:30:27 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:27.809005 13481 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:30:27.812361 13508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:27.813987 13481 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:30:27.814188 13481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "8e553d42833f4f3d92a937a3af295459"
format_stamp: "Formatted at 2026-06-19 06:30:24 on dist-test-slave-4t1f"
server_key: "188187d1b3316040fed99578f52ef1f7"
server_key_iv: "9634aa7278bf72ba9290b443e17be4b2"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:27.815294 13481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:27.844635 13481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:27.849102 13481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:27.849429 13481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:27.850472 13481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:27.851912 13481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:27.852031 13481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:27.852165 13481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:27.852280 13481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 06:30:27 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:27.893944 13481 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:44639
I20260619 06:30:27.894021 13620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:44639 every 8 connection(s)
I20260619 06:30:27.895603 13481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:30:27.905454 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13481
I20260619 06:30:27.906147 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.84:45085/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:44551
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:40169
--webserver_port=44081
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:30:27 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:27.964478 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:27.919002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40169 (local address 127.0.97.66:44037)
0619 06:30:27.943127 (+ 24125us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:27.943189 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:27.945696 (+ 2507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:27.946777 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:27.946789 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:27.947378 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:27.948137 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:27.948158 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.949648 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.949654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:27.951595 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:27.951609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.952607 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.952613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:27.952868 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:27.953686 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:27.953716 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:27.956390 (+ 2674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:27.960133 (+ 3743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:27.960141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:27.960154 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:27.960720 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:27.961212 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:27.961216 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:27.961220 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:27.961446 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:27.962164 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:27.962171 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:27.962511 (+ 340us) client_negotiation.cc:769] Sending connection context
0619 06:30:27.962787 (+ 276us) client_negotiation.cc:240] Negotiation successful
0619 06:30:27.963236 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23814,"thread_start_us":23552,"threads_started":1}
I20260619 06:30:27.965381 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:27.919551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44037 (local address 127.0.97.126:40169)
0619 06:30:27.942538 (+ 22987us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:27.942545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:27.943746 (+ 1201us) server_negotiation.cc:407] Connection header received
0619 06:30:27.946107 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:27.946113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:27.946195 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:27.946288 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:27.948344 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.949485 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.951759 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:27.952376 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:27.952514 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:27.956611 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:27.956617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:27.956622 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:27.956659 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:27.959984 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:27.960866 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:27.960872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:27.960877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:27.961022 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:27.961581 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:27.961586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:27.961589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:27.961947 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:27.962086 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:27.964665 (+ 2579us) server_negotiation.cc:299] Negotiation successful
0619 06:30:27.964935 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22834,"thread_start_us":22704,"threads_started":1}
I20260619 06:30:27.966801 13621 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40169
I20260619 06:30:27.967111 13621 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:27.967967 13621 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
I20260619 06:30:27.970048 13280 ts_manager.cc:194] Registered new tserver with Master: 8e553d42833f4f3d92a937a3af295459 (127.0.97.66:44639)
I20260619 06:30:27.972170 13280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:44037
Jun 19 06:30:28 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
W20260619 06:30:28.364629 13623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:28.364979 13623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:28.365207 13623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:28.376187 13623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:28.376319 13623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:28.376529 13623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:30:28.382403 13477 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:28.382802 13477 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
I20260619 06:30:28.384021 13280 ts_manager.cc:194] Registered new tserver with Master: e4c21bfad0974e0cb6e13bd20f7c83e1 (127.0.97.65:43627)
I20260619 06:30:28.385104 13280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:47863
I20260619 06:30:28.391491 13623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.84:45085/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:44551
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=44081
--tserver_master_addrs=127.0.97.126:40169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:28.393018 13623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:28.394618 13623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:28.411515 13631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:28.412026 13632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:28.420694 13623 server_base.cc:1034] running on GCE node
W20260619 06:30:28.421100 13634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:28.421890 13623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:28.425338 13623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:28.426584 13623 hybrid_clock.cc:648] HybridClock initialized: now 1781850628426517 us; error 63 us; skew 500 ppm
I20260619 06:30:28.426910 13623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:28 dist-test-slave-4t1f krb5kdc[12269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:28.430070 13623 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:30:28.433423 13623 webserver.cc:460] Webserver started at http://127.0.97.67:44081/ using document root <none> and password file <none>
I20260619 06:30:28.434108 13623 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:28.434269 13623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 06:30:28 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:28.451838 13623 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 06:30:28.454795 13642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:28.456286 13623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:30:28.456459 13623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "6815f4e1697b419bab93e6be1a06434e"
format_stamp: "Formatted at 2026-06-19 06:30:25 on dist-test-slave-4t1f"
server_key: "82884b059c12889f61458b7be1a092f2"
server_key_iv: "8f747385c750a7c1b7a24f708a96a5a8"
server_key_version: "kuduclusterkey@0"
I20260619 06:30:28.457504 13623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:28.470260 13623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:28.474169 13623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:28.474515 13623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:28.475523 13623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:28.476931 13623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:28.477049 13623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:28.477177 13623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:28.477278 13623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:28.531466 13623 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:44551
I20260619 06:30:28.531514 13754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:44551 every 8 connection(s)
I20260619 06:30:28.532935 13623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:30:28.537731 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13623
Jun 19 06:30:28 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:28.587980 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.536813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40169 (local address 127.0.97.67:38485)
0619 06:30:28.563032 (+ 26219us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:28.563091 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:28.567876 (+ 4785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:28.568889 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:28.568900 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:28.569452 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:28.570203 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.570225 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.571636 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.571642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.573644 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.573659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.575017 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.575025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.575320 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.576136 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:28.576169 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:28.578868 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:28.583107 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:28.583116 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:28.583137 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:28.583757 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:28.584189 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:28.584195 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:28.584200 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:28.584433 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:28.585218 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:28.585229 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:28.585656 (+ 427us) client_negotiation.cc:769] Sending connection context
0619 06:30:28.585976 (+ 320us) client_negotiation.cc:240] Negotiation successful
0619 06:30:28.586537 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25856,"thread_start_us":28539,"threads_started":1}
I20260619 06:30:28.589201 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.536040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:38485 (local address 127.0.97.126:40169)
0619 06:30:28.561869 (+ 25829us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:28.561875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:28.565105 (+ 3230us) server_negotiation.cc:407] Connection header received
0619 06:30:28.568240 (+ 3135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:28.568246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:28.568331 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:28.568419 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:28.570398 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.571474 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.573826 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.574555 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.574684 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.579125 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:28.579131 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:28.579137 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:28.579172 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:28.582923 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:28.583883 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:28.583890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:28.583898 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:28.584062 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:28.584572 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:28.584580 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:28.584587 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:28.584962 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:28.585145 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:28.588466 (+ 3321us) server_negotiation.cc:299] Negotiation successful
0619 06:30:28.588753 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":25703,"thread_start_us":31155,"threads_started":1}
I20260619 06:30:28.591189 13755 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40169
I20260619 06:30:28.591485 13755 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:28.592208 13755 heartbeater.cc:511] Master 127.0.97.126:40169 requested a full tablet report, sending...
I20260619 06:30:28.593423 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.538865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.0.97.126:40169)
0619 06:30:28.566759 (+ 27894us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:28.566766 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:28.566800 (+ 34us) server_negotiation.cc:407] Connection header received
0619 06:30:28.566855 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:28.566859 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:28.566962 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:28.567080 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:28.574123 (+ 7043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.576162 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.578959 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.579699 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.579837 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.583811 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:28.583820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:28.583824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:28.583863 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:28.587218 (+ 3355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:28.588078 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:28.588082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:28.588085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:28.588225 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:28.588657 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:28.588664 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:28.588670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:28.588996 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:28.589324 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:28.591023 (+ 1699us) server_negotiation.cc:299] Negotiation successful
0619 06:30:28.592997 (+ 1974us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":27772,"thread_start_us":23025,"threads_started":1}
I20260619 06:30:28.593757 13280 ts_manager.cc:194] Registered new tserver with Master: 6815f4e1697b419bab93e6be1a06434e (127.0.97.67:44551)
I20260619 06:30:28.594862 13280 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:38485
I20260619 06:30:28.596184 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:30:28.616966 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.601549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.0.97.126:40169)
0619 06:30:28.602478 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:28.602487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:28.603004 (+ 517us) server_negotiation.cc:407] Connection header received
0619 06:30:28.603319 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:28.603324 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:28.603401 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:28.603467 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:28.605034 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.605740 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.607073 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.607631 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.607754 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.611221 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:28.611227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:28.611231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:28.611268 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:28.614121 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:28.614780 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:28.614788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:28.614795 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:28.614937 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:28.615395 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:28.615401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:28.615405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:28.615698 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:28.615927 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:28.616369 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 06:30:28.616585 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738}
I20260619 06:30:28.623502 13279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53172:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:30:28.633797 13279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 19 06:30:28 dist-test-slave-4t1f krb5kdc[12269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for HTTP/127.0.97.84@KRBTEST.COM
I20260619 06:30:28.707059 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.667929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:44639 (local address 127.0.0.1:58232)
0619 06:30:28.695365 (+ 27436us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:28.695385 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:28.695605 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:28.696693 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:28.696698 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:28.696724 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:28.697111 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.697124 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.699845 (+ 2721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.699854 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.702282 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.702299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.705506 (+ 3207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.705525 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.705725 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.705782 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 06:30:28.705900 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 06:30:28.706085 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27305,"thread_start_us":27149,"threads_started":1}
I20260619 06:30:28.707337 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.669227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58232 (local address 127.0.97.66:44639)
0619 06:30:28.684492 (+ 15265us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:28.684503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:28.695493 (+ 10990us) server_negotiation.cc:407] Connection header received
0619 06:30:28.695840 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:28.695849 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:28.696183 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:28.696398 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:28.697349 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.699657 (+ 2308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.703450 (+ 3793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.704999 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.705111 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.705312 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:28.706826 (+ 1514us) server_negotiation.cc:299] Negotiation successful
0619 06:30:28.707006 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15067,"thread_start_us":14949,"threads_started":1}
I20260619 06:30:28.710843 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.668656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40448 (local address 127.0.97.67:44551)
0619 06:30:28.669413 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:28.669422 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:28.685338 (+ 15916us) server_negotiation.cc:407] Connection header received
0619 06:30:28.685805 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:28.685814 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:28.686074 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:28.686249 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:28.688864 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.690551 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.707801 (+ 17250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.709328 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.709448 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.709686 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:28.710316 (+ 630us) server_negotiation.cc:299] Negotiation successful
0619 06:30:28.710480 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":701,"threads_started":1}
I20260619 06:30:28.714280 13556 tablet_service.cc:1467] Processing CreateTablet for tablet 5173e72d030e4c19b5983c72091580bf (DEFAULT_TABLE table=test-table [id=db830f6e2d604e12bc083a61f88eea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:28.715838 13556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5173e72d030e4c19b5983c72091580bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:28.716604 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.692122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43627 (local address 127.0.0.1:46624)
0619 06:30:28.701625 (+ 9503us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:28.701643 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:28.701750 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:28.704570 (+ 2820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:28.704575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:28.704599 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:28.704929 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.704940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.706932 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.706938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.710612 (+ 3674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.710626 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.715846 (+ 5220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.715855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.716022 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.716053 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:30:28.716139 (+ 86us) client_negotiation.cc:240] Negotiation successful
0619 06:30:28.716273 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9379,"thread_start_us":9685,"threads_started":1}
I20260619 06:30:28.717245 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.667784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44551 (local address 127.0.0.1:40448)
0619 06:30:28.685163 (+ 17379us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:28.685213 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:28.685531 (+ 318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:28.687128 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:28.687136 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:28.687181 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:28.687658 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.687672 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.690749 (+ 3077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.690754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.693193 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:28.693208 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.709922 (+ 16714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.709928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:28.710121 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.710155 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:30:28.716789 (+ 6634us) client_negotiation.cc:240] Negotiation successful
0619 06:30:28.716930 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17038,"thread_start_us":16903,"threads_started":1}
I20260619 06:30:28.718024 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:28.686176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.0.97.65:43627)
0619 06:30:28.703750 (+ 17574us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:28.703759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:28.703787 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:30:28.703886 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:28.703894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:28.704158 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:28.704345 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:28.705328 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.706796 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.711375 (+ 4579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:28.715306 (+ 3931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:28.715441 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:28.715687 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:28.717509 (+ 1822us) server_negotiation.cc:299] Negotiation successful
0619 06:30:28.717695 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17395,"thread_start_us":18883,"threads_started":1}
I20260619 06:30:28.720930 13690 tablet_service.cc:1467] Processing CreateTablet for tablet 5173e72d030e4c19b5983c72091580bf (DEFAULT_TABLE table=test-table [id=db830f6e2d604e12bc083a61f88eea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:28.722355 13690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5173e72d030e4c19b5983c72091580bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:28.728616 13412 tablet_service.cc:1467] Processing CreateTablet for tablet 5173e72d030e4c19b5983c72091580bf (DEFAULT_TABLE table=test-table [id=db830f6e2d604e12bc083a61f88eea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:28.730020 13412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5173e72d030e4c19b5983c72091580bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:28.776871 13778 tablet_bootstrap.cc:492] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Bootstrap starting.
I20260619 06:30:28.780516 13777 tablet_bootstrap.cc:492] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Bootstrap starting.
I20260619 06:30:28.781518 13779 tablet_bootstrap.cc:492] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Bootstrap starting.
I20260619 06:30:28.785425 13778 tablet_bootstrap.cc:654] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:28.785444 13777 tablet_bootstrap.cc:654] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:28.786702 13777 log.cc:824] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:28.788810 13779 tablet_bootstrap.cc:654] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:28.795811 13779 log.cc:824] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:28.797252 13778 log.cc:824] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:28.798569 13777 tablet_bootstrap.cc:492] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: No bootstrap required, opened a new log
I20260619 06:30:28.798892 13777 ts_tablet_manager.cc:1397] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.004s
I20260619 06:30:28.809456 13777 raft_consensus.cc:348] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:28.810034 13777 raft_consensus.cc:374] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:28.810171 13777 raft_consensus.cc:729] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c21bfad0974e0cb6e13bd20f7c83e1, State: Initialized, Role: FOLLOWER
I20260619 06:30:28.810977 13777 consensus_queue.cc:260] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:28.814218 13779 tablet_bootstrap.cc:492] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: No bootstrap required, opened a new log
I20260619 06:30:28.817170 13779 ts_tablet_manager.cc:1397] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.000s
I20260619 06:30:28.829272 13778 tablet_bootstrap.cc:492] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: No bootstrap required, opened a new log
I20260619 06:30:28.829627 13778 ts_tablet_manager.cc:1397] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.005s
I20260619 06:30:28.829814 13477 heartbeater.cc:503] Master 127.0.97.126:40169 was elected leader, sending a full tablet report...
I20260619 06:30:28.830508 13777 ts_tablet_manager.cc:1428] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Time spent starting tablet: real 0.031s user 0.018s sys 0.010s
I20260619 06:30:28.839840 13778 raft_consensus.cc:348] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:28.840370 13778 raft_consensus.cc:374] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:28.840497 13778 raft_consensus.cc:729] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e553d42833f4f3d92a937a3af295459, State: Initialized, Role: FOLLOWER
I20260619 06:30:28.841253 13778 consensus_queue.cc:260] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:28.841696 13779 raft_consensus.cc:348] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:28.842203 13779 raft_consensus.cc:374] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:28.842360 13779 raft_consensus.cc:729] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6815f4e1697b419bab93e6be1a06434e, State: Initialized, Role: FOLLOWER
I20260619 06:30:28.842957 13779 consensus_queue.cc:260] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
W20260619 06:30:28.849700 13478 tablet.cc:2369] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:30:28.850188 13621 heartbeater.cc:503] Master 127.0.97.126:40169 was elected leader, sending a full tablet report...
I20260619 06:30:28.850981 13778 ts_tablet_manager.cc:1428] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Time spent starting tablet: real 0.021s user 0.008s sys 0.010s
I20260619 06:30:28.851938 13755 heartbeater.cc:503] Master 127.0.97.126:40169 was elected leader, sending a full tablet report...
I20260619 06:30:28.852555 13779 ts_tablet_manager.cc:1428] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Time spent starting tablet: real 0.034s user 0.019s sys 0.000s
W20260619 06:30:28.903630 13622 tablet.cc:2369] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:30:29.046340 13756 tablet.cc:2369] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:30:29.190506 13784 raft_consensus.cc:482] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:30:29.190891 13784 raft_consensus.cc:504] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:29.192582 13784 leader_election.cc:288] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6815f4e1697b419bab93e6be1a06434e (127.0.97.67:44551), e4c21bfad0974e0cb6e13bd20f7c83e1 (127.0.97.65:43627)
I20260619 06:30:29.200930 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.192970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:58627 (local address 127.0.97.67:44551)
0619 06:30:29.193270 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:29.193277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:29.193751 (+ 474us) server_negotiation.cc:407] Connection header received
0619 06:30:29.194069 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:29.194074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:29.194156 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:29.194223 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:29.195189 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.196474 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.199059 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.200046 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.200180 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.200278 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:29.200553 (+ 275us) server_negotiation.cc:299] Negotiation successful
0619 06:30:29.200678 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:30:29.201097 13793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.192880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:44551 (local address 127.0.97.66:58627)
0619 06:30:29.193618 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:29.193639 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:29.193883 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:29.194434 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:29.194438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:29.194474 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:29.195001 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:29.195015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.196627 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.196676 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:29.198906 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:29.198922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.200234 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.200238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:29.200366 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.200388 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:30:29.200502 (+ 114us) client_negotiation.cc:240] Negotiation successful
0619 06:30:29.200585 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":1,"thread_start_us":372,"threads_started":1}
I20260619 06:30:29.201622 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.194210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46343 (local address 127.0.97.65:43627)
0619 06:30:29.194475 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:29.194481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:29.195228 (+ 747us) server_negotiation.cc:407] Connection header received
0619 06:30:29.195362 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:29.195367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:29.195446 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:29.195531 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:29.196256 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.197293 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.199484 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.200632 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.200748 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.200840 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:29.201271 (+ 431us) server_negotiation.cc:299] Negotiation successful
0619 06:30:29.201402 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:30:29.201683 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.193707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:43627 (local address 127.0.97.66:46343)
0619 06:30:29.195108 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:29.195124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:29.195242 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:29.195665 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:29.195670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:29.195699 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:29.196011 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:29.196021 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.197429 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.197434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:29.199338 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:29.199348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.200858 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.200864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:29.201100 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.201140 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:30:29.201245 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:30:29.201367 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"thread_start_us":1060,"threads_started":1}
I20260619 06:30:29.202144 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5173e72d030e4c19b5983c72091580bf" candidate_uuid: "8e553d42833f4f3d92a937a3af295459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f4e1697b419bab93e6be1a06434e" is_pre_election: true
I20260619 06:30:29.202557 13710 raft_consensus.cc:2454] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e553d42833f4f3d92a937a3af295459 in term 0.
I20260619 06:30:29.203114 13512 leader_election.cc:302] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6815f4e1697b419bab93e6be1a06434e, 8e553d42833f4f3d92a937a3af295459; no voters:
I20260619 06:30:29.203517 13784 raft_consensus.cc:2781] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:30:29.203663 13784 raft_consensus.cc:482] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:30:29.203783 13784 raft_consensus.cc:3037] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:29.203713 13432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5173e72d030e4c19b5983c72091580bf" candidate_uuid: "8e553d42833f4f3d92a937a3af295459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" is_pre_election: true
I20260619 06:30:29.204211 13432 raft_consensus.cc:2454] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e553d42833f4f3d92a937a3af295459 in term 0.
I20260619 06:30:29.206238 13784 raft_consensus.cc:504] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:29.206779 13784 leader_election.cc:288] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [CANDIDATE]: Term 1 election: Requested vote from peers 6815f4e1697b419bab93e6be1a06434e (127.0.97.67:44551), e4c21bfad0974e0cb6e13bd20f7c83e1 (127.0.97.65:43627)
I20260619 06:30:29.207257 13710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5173e72d030e4c19b5983c72091580bf" candidate_uuid: "8e553d42833f4f3d92a937a3af295459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6815f4e1697b419bab93e6be1a06434e"
I20260619 06:30:29.207304 13432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5173e72d030e4c19b5983c72091580bf" candidate_uuid: "8e553d42833f4f3d92a937a3af295459" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1"
I20260619 06:30:29.207432 13710 raft_consensus.cc:3037] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:29.207546 13432 raft_consensus.cc:3037] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:29.209796 13710 raft_consensus.cc:2454] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e553d42833f4f3d92a937a3af295459 in term 1.
I20260619 06:30:29.209970 13432 raft_consensus.cc:2454] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e553d42833f4f3d92a937a3af295459 in term 1.
I20260619 06:30:29.210182 13512 leader_election.cc:302] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6815f4e1697b419bab93e6be1a06434e, 8e553d42833f4f3d92a937a3af295459; no voters:
I20260619 06:30:29.210477 13784 raft_consensus.cc:2781] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:30:29.210928 13784 raft_consensus.cc:686] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 1 LEADER]: Becoming Leader. State: Replica: 8e553d42833f4f3d92a937a3af295459, State: Running, Role: LEADER
I20260619 06:30:29.211453 13784 consensus_queue.cc:237] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } }
I20260619 06:30:29.215885 13279 catalog_manager.cc:5495] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e553d42833f4f3d92a937a3af295459 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "8e553d42833f4f3d92a937a3af295459" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e553d42833f4f3d92a937a3af295459" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 44639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 } health_report { overall_health: UNKNOWN } } }
I20260619 06:30:29.274258 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.267976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.97.66:44639)
0619 06:30:29.268650 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:29.268655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:29.268672 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:30:29.268760 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:29.268763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:29.268834 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:29.268998 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:30:29.269454 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.270164 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.271578 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.272383 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.272499 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.273034 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:30:29.273631 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:30:29.273806 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:29.273858 (+ 52us) server_negotiation.cc:299] Negotiation successful
0619 06:30:29.273934 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":357,"threads_started":1}
I20260619 06:30:29.282073 13432 raft_consensus.cc:1261] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 1 FOLLOWER]: Refusing update from remote peer 8e553d42833f4f3d92a937a3af295459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:29.282078 13710 raft_consensus.cc:1261] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 1 FOLLOWER]: Refusing update from remote peer 8e553d42833f4f3d92a937a3af295459: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:29.282989 13795 consensus_queue.cc:1035] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6815f4e1697b419bab93e6be1a06434e" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 44551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:29.283393 13784 consensus_queue.cc:1035] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4c21bfad0974e0cb6e13bd20f7c83e1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 43627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:29.310279 13280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:30:29.310952 13280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53172:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:30:29.315701 13280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 500e55da023b4d3487f4360b663f2ba8: Sending DeleteTablet for 3 replicas of tablet 5173e72d030e4c19b5983c72091580bf
I20260619 06:30:29.317134 13690 tablet_service.cc:1514] Processing DeleteTablet for tablet 5173e72d030e4c19b5983c72091580bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:29 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40448
I20260619 06:30:29.317989 13412 tablet_service.cc:1514] Processing DeleteTablet for tablet 5173e72d030e4c19b5983c72091580bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:29 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:46624
I20260619 06:30:29.318379 13556 tablet_service.cc:1514] Processing DeleteTablet for tablet 5173e72d030e4c19b5983c72091580bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:29 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:58232
I20260619 06:30:29.318786 13813 tablet_replica.cc:330] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: stopping tablet replica
I20260619 06:30:29.319293 13813 raft_consensus.cc:2229] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:30:29.319855 13813 raft_consensus.cc:2258] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:29.321810 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13317
I20260619 06:30:29.324788 13813 ts_tablet_manager.cc:1905] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:29.327924 13814 tablet_replica.cc:330] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: stopping tablet replica
I20260619 06:30:29.328297 13814 raft_consensus.cc:2229] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:30:29.328822 13814 raft_consensus.cc:2258] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:29.330804 13814 ts_tablet_manager.cc:1905] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:29.336376 13813 ts_tablet_manager.cc:1918] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:29.336642 13813 log.cc:1196] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/5173e72d030e4c19b5983c72091580bf
I20260619 06:30:29.337975 13813 ts_tablet_manager.cc:1939] T 5173e72d030e4c19b5983c72091580bf P e4c21bfad0974e0cb6e13bd20f7c83e1: Deleting consensus metadata
I20260619 06:30:29.340413 13814 ts_tablet_manager.cc:1918] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:29.340562 13814 log.cc:1196] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/5173e72d030e4c19b5983c72091580bf
I20260619 06:30:29.340406 13267 catalog_manager.cc:4841] TS e4c21bfad0974e0cb6e13bd20f7c83e1 (127.0.97.65:43627): tablet 5173e72d030e4c19b5983c72091580bf (table test-table [id=db830f6e2d604e12bc083a61f88eea7a]) successfully deleted
I20260619 06:30:29.341100 13814 ts_tablet_manager.cc:1939] T 5173e72d030e4c19b5983c72091580bf P 8e553d42833f4f3d92a937a3af295459: Deleting consensus metadata
I20260619 06:30:29.342259 13812 tablet_replica.cc:330] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: stopping tablet replica
I20260619 06:30:29.342731 13812 raft_consensus.cc:2229] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:30:29.342693 13264 catalog_manager.cc:4841] TS 8e553d42833f4f3d92a937a3af295459 (127.0.97.66:44639): tablet 5173e72d030e4c19b5983c72091580bf (table test-table [id=db830f6e2d604e12bc083a61f88eea7a]) successfully deleted
I20260619 06:30:29.345019 13812 raft_consensus.cc:2258] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:29.346985 13812 ts_tablet_manager.cc:1905] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:29.354199 13812 ts_tablet_manager.cc:1918] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:29.354378 13812 log.cc:1196] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/5173e72d030e4c19b5983c72091580bf
I20260619 06:30:29.354885 13812 ts_tablet_manager.cc:1939] T 5173e72d030e4c19b5983c72091580bf P 6815f4e1697b419bab93e6be1a06434e: Deleting consensus metadata
I20260619 06:30:29.356798 13267 catalog_manager.cc:4841] TS 6815f4e1697b419bab93e6be1a06434e (127.0.97.67:44551): tablet 5173e72d030e4c19b5983c72091580bf (table test-table [id=db830f6e2d604e12bc083a61f88eea7a]) successfully deleted
I20260619 06:30:29.365119 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.323315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.0.97.65:43627)
0619 06:30:29.323483 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:29.323491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:29.341491 (+ 18000us) server_negotiation.cc:407] Connection header received
0619 06:30:29.341692 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:29.341698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:29.341831 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:29.341978 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:29.344606 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.345784 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.348442 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.349196 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.349299 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.355274 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:29.355323 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:29.355346 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:29.355405 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:29.360101 (+ 4696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:29.361022 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:29.361034 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:29.361039 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:29.361255 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:29.361946 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:29.361953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:29.361959 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:29.362580 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:29.362912 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:29.364303 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0619 06:30:29.364592 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:30:29.366104 13472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:29.463788 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13317
I20260619 06:30:29.483009 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13481
I20260619 06:30:29.498981 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.483782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.0.97.66:44639)
0619 06:30:29.483964 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:29.483968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:29.484022 (+ 54us) server_negotiation.cc:407] Connection header received
0619 06:30:29.484197 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:29.484201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:29.484272 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:29.484361 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:29.486222 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.486986 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.488872 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.489476 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.489588 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.492754 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:29.492768 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:29.492783 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:29.492815 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:29.496020 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:29.496855 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:29.496862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:29.496864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:29.496981 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:29.497506 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:29.497509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:29.497511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:29.497858 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:29.498115 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:29.498445 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 06:30:29.498632 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:30:29.499719 13616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:29.612453 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13481
I20260619 06:30:29.630856 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13623
I20260619 06:30:29.644424 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:29.631640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40452 (local address 127.0.97.67:44551)
0619 06:30:29.631809 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:29.631817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:29.631889 (+ 72us) server_negotiation.cc:407] Connection header received
0619 06:30:29.632095 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:29.632105 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:29.632176 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:29.632308 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:29.633704 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.634426 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.635755 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:29.636154 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:29.636285 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:29.638807 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:29.638827 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:29.638843 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:29.638878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:29.641762 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:29.642430 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:29.642443 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:29.642445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:29.642568 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:29.642922 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:29.642928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:29.642931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:29.643281 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:29.643593 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:29.643879 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 06:30:29.644089 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:30:29.645198 13750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:29.735867 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13623
I20260619 06:30:29.752292 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13247
I20260619 06:30:29.753659 13310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:29.866050 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13247
I20260619 06:30:29.887275 389 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 06:30:30.222033 389 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 06:30:30.222288 389 mini_ranger.cc:66] Stopping Ranger...
I20260619 06:30:30.567797 389 mini_ranger.cc:68] Stopped Ranger
2026-06-19 06:30:30.568 UTC [12299] LOG: received smart shutdown request
2026-06-19 06:30:30.571 UTC [12299] LOG: background worker "logical replication launcher" (PID 12308) exited with exit code 1
2026-06-19 06:30:30.571 UTC [12303] LOG: shutting down
2026-06-19 06:30:30.728 UTC [12299] LOG: database system is shut down
2026-06-19T06:30:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51385 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:30:30 dist-test-slave-4t1f krb5kdc[13828](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:30:30 dist-test-slave-4t1f krb5kdc[13828](info): set up 2 sockets
Jun 19 06:30:30 dist-test-slave-4t1f krb5kdc[13828](info): commencing operation
krb5kdc: starting...
W20260619 06:30:31.953390 389 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:30:32 dist-test-slave-4t1f krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:30:32.175872 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41493
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:41493,127.0.97.125:41299,127.0.97.124:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:32.498606 13841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:32.498966 13841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:32.499145 13841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:32.506371 13841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:32.506517 13841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:32.506639 13841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:32.506742 13841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:32.506832 13841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:32.517053 13841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:41493,127.0.97.125:41299,127.0.97.124:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:32.518440 13841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:32.519753 13841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:32.531315 13849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:32.531245 13847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:32.531353 13846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:32.532431 13841 server_base.cc:1034] running on GCE node
I20260619 06:30:32.533202 13841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:32.534487 13841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:32.535715 13841 hybrid_clock.cc:648] HybridClock initialized: now 1781850632535678 us; error 57 us; skew 500 ppm
I20260619 06:30:32.536056 13841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:32 dist-test-slave-4t1f krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:32.539193 13841 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:30:32.540503 13841 webserver.cc:460] Webserver started at http://127.0.97.126:45383/ using document root <none> and password file <none>
I20260619 06:30:32.541232 13841 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:32.541330 13841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:32.541565 13841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:30:32.544301 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "ad6ba9c439a24b469261a608dd382724"
format_stamp: "Formatted at 2026-06-19 06:30:32 on dist-test-slave-4t1f"
server_key: "1f1fba3f66557f926b2b020e449a0f0e"
server_key_iv: "49268a76ff4b410b86922f8ce6b3ecf6"
server_key_version: "encryptionkey@0"
I20260619 06:30:32.545161 13841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "ad6ba9c439a24b469261a608dd382724"
format_stamp: "Formatted at 2026-06-19 06:30:32 on dist-test-slave-4t1f"
server_key: "1f1fba3f66557f926b2b020e449a0f0e"
server_key_iv: "49268a76ff4b410b86922f8ce6b3ecf6"
server_key_version: "encryptionkey@0"
I20260619 06:30:32.549952 13841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:30:32.553112 13856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:32.554330 13841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:30:32.554463 13841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "ad6ba9c439a24b469261a608dd382724"
format_stamp: "Formatted at 2026-06-19 06:30:32 on dist-test-slave-4t1f"
server_key: "1f1fba3f66557f926b2b020e449a0f0e"
server_key_iv: "49268a76ff4b410b86922f8ce6b3ecf6"
server_key_version: "encryptionkey@0"
I20260619 06:30:32.554620 13841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:32.569151 13841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:32.573200 13841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:32.573405 13841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:32.591321 13841 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:41493
I20260619 06:30:32.591360 13907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:41493 every 8 connection(s)
I20260619 06:30:32.592640 13841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:30:32.595865 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13841
I20260619 06:30:32.596091 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:30:32.596684 13908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:32.596686 389 external_mini_cluster.cc:1428] Setting key 353590154c7f55b8410128246eb02524
I20260619 06:30:32.602975 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
Jun 19 06:30:32 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:32.625813 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.607453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.0.97.126:41493)
0619 06:30:32.608136 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:32.608144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:32.608199 (+ 55us) server_negotiation.cc:407] Connection header received
0619 06:30:32.609349 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:32.609354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:32.609576 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:32.609776 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:32.610979 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:32.612165 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:32.613596 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:32.614121 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:32.614322 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:32.619077 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:32.619085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:32.619096 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:32.619122 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:32.621839 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:32.622521 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:32.622529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:32.622531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:32.622661 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:32.623084 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:32.623094 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:32.623097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:32.623674 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:32.623888 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:32.624448 (+ 560us) server_negotiation.cc:299] Negotiation successful
0619 06:30:32.624811 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":481,"thread_start_us":314,"threads_started":1}
I20260619 06:30:32.625850 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.606175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41493 (local address 127.0.0.1:48778)
0619 06:30:32.606950 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.606987 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:32.608927 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:32.609788 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:32.609794 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:32.610472 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:32.610786 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:32.610802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:32.612320 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:32.612327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:32.613458 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:32.613469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:32.614368 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:32.614375 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:32.614626 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:32.615303 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:32.615324 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:32.618857 (+ 3533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:32.621976 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:32.621988 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:32.621999 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:32.622368 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:32.622775 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:32.622781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:32.622787 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:32.622932 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:32.623836 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:32.623842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:32.624025 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 06:30:32.624235 (+ 210us) client_negotiation.cc:240] Negotiation successful
0619 06:30:32.624667 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":52,"thread_start_us":344,"threads_started":1}
I20260619 06:30:32.628925 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:30:32.630147 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.629678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57848)
0619 06:30:32.629871 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.629998 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260619 06:30:32.630432 13859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
I20260619 06:30:32.632148 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.631694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57858)
0619 06:30:32.631870 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.631960 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:30:32.632376 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:41299: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260619 06:30:32.644543 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:41299
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:41493,127.0.97.125:41299,127.0.97.124:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:30:32.681725 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } attempt: 1
I20260619 06:30:32.682988 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.682441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57860)
0619 06:30:32.682683 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.682813 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:30:32.684053 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.683668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57870)
0619 06:30:32.683816 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.683901 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:30:32.684305 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:41299: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
I20260619 06:30:32.752583 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } attempt: 2
I20260619 06:30:32.753891 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.753314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57880)
0619 06:30:32.753594 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.753708 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:30:32.754979 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.754576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57890)
0619 06:30:32.754730 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.754831 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:30:32.755201 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:41299: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
I20260619 06:30:32.846483 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } attempt: 3
I20260619 06:30:32.847856 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.847195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57894)
0619 06:30:32.847501 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.847655 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260619 06:30:32.848910 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.848511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57898)
0619 06:30:32.848668 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.848752 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:30:32.849131 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:41299: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
W20260619 06:30:32.960290 13915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:32.960554 13915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:32.960734 13915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:32.968145 13915 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:32.968241 13915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:32.968331 13915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:32.968434 13915 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:32.968514 13915 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:32.978333 13915 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:41493,127.0.97.125:41299,127.0.97.124:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:32.979588 13915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:32.980870 13915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:30:32.992431 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } attempt: 4
W20260619 06:30:32.992604 13920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:32.992750 13923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:32.993641 13921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:32.993872 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.993230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57912)
0619 06:30:32.993565 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.993695 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
I20260619 06:30:32.993824 13915 server_base.cc:1034] running on GCE node
I20260619 06:30:32.994882 13915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:32.995064 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:32.994579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57916)
0619 06:30:32.994771 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:32.994895 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 06:30:32.995309 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:41299: Network error: Client connection negotiation failed: client connection to 127.0.97.125:41299: connect: Connection refused (error 111)
I20260619 06:30:32.996443 13915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:32.997668 13915 hybrid_clock.cc:648] HybridClock initialized: now 1781850632997612 us; error 37 us; skew 500 ppm
I20260619 06:30:32.998063 13915 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:32 dist-test-slave-4t1f krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:33.001336 13915 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260619 06:30:33.002893 13915 webserver.cc:460] Webserver started at http://127.0.97.125:43227/ using document root <none> and password file <none>
I20260619 06:30:33.003659 13915 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:33.003805 13915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:33.004138 13915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:30:33.007309 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data/instance:
uuid: "67265de3b4f24403b0ed3b2f83c2a22d"
format_stamp: "Formatted at 2026-06-19 06:30:32 on dist-test-slave-4t1f"
server_key: "a25f80ce186bff58ce07ac60c0575d02"
server_key_iv: "02a8a65a80470950a351bcb8e8318f0c"
server_key_version: "encryptionkey@0"
I20260619 06:30:33.008170 13915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/wal/instance:
uuid: "67265de3b4f24403b0ed3b2f83c2a22d"
format_stamp: "Formatted at 2026-06-19 06:30:32 on dist-test-slave-4t1f"
server_key: "a25f80ce186bff58ce07ac60c0575d02"
server_key_iv: "02a8a65a80470950a351bcb8e8318f0c"
server_key_version: "encryptionkey@0"
I20260619 06:30:33.013588 13915 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:30:33.017140 13930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:33.018457 13915 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:30:33.018642 13915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/wal
uuid: "67265de3b4f24403b0ed3b2f83c2a22d"
format_stamp: "Formatted at 2026-06-19 06:30:32 on dist-test-slave-4t1f"
server_key: "a25f80ce186bff58ce07ac60c0575d02"
server_key_iv: "02a8a65a80470950a351bcb8e8318f0c"
server_key_version: "encryptionkey@0"
I20260619 06:30:33.018826 13915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:33.037736 13915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:33.041536 13915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:33.041836 13915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:33.060243 13915 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:41299
I20260619 06:30:33.060281 13981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:41299 every 8 connection(s)
I20260619 06:30:33.061713 13915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
I20260619 06:30:33.064064 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13915
I20260619 06:30:33.064247 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-1/wal/instance
I20260619 06:30:33.064792 389 external_mini_cluster.cc:1428] Setting key 8875aae43241d572e42d864aea7d7728
I20260619 06:30:33.065970 13982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:33.075160 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:33.096906 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.077970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.0.97.126:41493)
0619 06:30:33.078150 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.078155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.079509 (+ 1354us) server_negotiation.cc:407] Connection header received
0619 06:30:33.081431 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.081437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.081532 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.081649 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.083270 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.083944 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.085534 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.085981 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.086124 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.089861 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.089871 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.089879 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.089927 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.093683 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.094488 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.094496 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.094502 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.094650 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.095079 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.095086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.095092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.095498 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.095656 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.096514 (+ 858us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.096675 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:30:33.098459 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.078532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41493 (local address 127.0.0.1:48790)
0619 06:30:33.079330 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.079368 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.081161 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.081883 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.081891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.082522 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.083095 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.083113 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.084111 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.084115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.085382 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.085394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.086215 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.086222 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.086456 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.087214 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.087236 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.089624 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.093841 (+ 4217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.093848 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.093860 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.094274 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.094768 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.094776 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.094778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.094957 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.095620 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.095626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.096098 (+ 472us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.096412 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.096968 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":358,"threads_started":1}
I20260619 06:30:33.100198 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:30:33.114614 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.100959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57922)
0619 06:30:33.101168 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.101187 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.101282 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.102261 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.102265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.102695 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.102818 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.102826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.104169 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.104175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.104987 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.104997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.105946 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.105951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.106144 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.106759 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.106775 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.108686 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.111960 (+ 3274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.111965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.111970 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.112340 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.112783 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.112788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.112791 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.112954 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.113649 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.113654 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.113789 (+ 135us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.113935 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.114203 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:30:33.114717 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.101252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57922 (local address 127.0.97.125:41299)
0619 06:30:33.101880 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.101885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.101910 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:30:33.101950 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.101955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.102121 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.102271 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.102971 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.104022 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.105154 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.105797 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.105939 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.108885 (+ 2946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.108895 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.108909 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.108938 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.111809 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.112473 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.112489 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.112491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.112669 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.113135 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.113140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.113143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.113480 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.113707 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.114080 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.114315 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":491,"thread_start_us":359,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260619 06:30:33.117147 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:44663
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:41493,127.0.97.125:41299,127.0.97.124:44663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:30:33.117355 13982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:30:33.118496 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.118041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43676)
0619 06:30:33.118222 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.118348 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 06:30:33.118754 13932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.120438 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.120089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43680)
0619 06:30:33.120249 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.120332 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:30:33.120695 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.169994 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 1
I20260619 06:30:33.171236 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.170726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43684)
0619 06:30:33.170956 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.171057 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260619 06:30:33.172326 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.171958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43696)
0619 06:30:33.172113 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.172210 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 06:30:33.172574 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.240790 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 2
I20260619 06:30:33.242035 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.241581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43706)
0619 06:30:33.241782 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.241869 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 06:30:33.243058 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.242685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43708)
0619 06:30:33.242840 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.242923 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:30:33.243247 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.294576 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } attempt: 5
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:30:33.310449 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.295283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57928)
0619 06:30:33.295525 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.295548 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.295681 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.296034 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.296039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.296542 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.296765 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.296778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.297826 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.297834 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.299516 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.299535 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.300314 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.300322 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.300557 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.301341 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.301382 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.304054 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.308202 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.308206 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.308209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.308579 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.308972 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.308975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.308977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.309173 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.309755 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.309759 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.309896 (+ 137us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.310008 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.310207 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 06:30:33.310680 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.295519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57928 (local address 127.0.97.125:41299)
0619 06:30:33.295692 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.295696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.295717 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:30:33.295779 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.295782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.295854 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.295939 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.296938 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.297673 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.299685 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.300128 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.300229 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.304276 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.304284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.304292 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.304342 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.308030 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.308701 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.308707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.308713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.308859 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.309278 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.309282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.309285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.309645 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.309788 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.310148 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.310383 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:30:33.311793 13908 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:30:33.312777 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.312408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43714)
0619 06:30:33.312571 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.312644 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:30:33.313704 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.313387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43718)
0619 06:30:33.313529 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.313614 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:30:33.313903 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.334444 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 3
I20260619 06:30:33.335580 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.335101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43726)
0619 06:30:33.335331 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.335422 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:30:33.336678 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.336256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43740)
0619 06:30:33.336479 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.336564 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260619 06:30:33.336933 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.365132 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 1
I20260619 06:30:33.366250 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.365823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43748)
0619 06:30:33.366007 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.366101 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:30:33.367205 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.366887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43750)
0619 06:30:33.367030 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.367107 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:30:33.367419 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.435631 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 2
I20260619 06:30:33.436748 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.436286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43762)
0619 06:30:33.436510 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.436592 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:30:33.437723 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.437393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43776)
0619 06:30:33.437547 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.437621 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:30:33.437912 13908 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
W20260619 06:30:33.439596 13989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:33.439913 13989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:33.440073 13989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:33.447186 13989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:33.447331 13989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:33.447434 13989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:33.447536 13989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:33.447623 13989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:33.457187 13989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:41493,127.0.97.125:41299,127.0.97.124:44663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:44663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:33.458509 13989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:33.459810 13989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:33.471478 13994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:33.471522 13995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:33.471553 13997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:33.472296 13989 server_base.cc:1034] running on GCE node
I20260619 06:30:33.473250 13989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:33.474686 13989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:33.475920 13989 hybrid_clock.cc:648] HybridClock initialized: now 1781850633475860 us; error 52 us; skew 500 ppm
I20260619 06:30:33.476256 13989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:33.479552 13989 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260619 06:30:33.480146 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 4
I20260619 06:30:33.481249 13989 webserver.cc:460] Webserver started at http://127.0.97.124:43141/ using document root <none> and password file <none>
I20260619 06:30:33.481281 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.480767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43782)
0619 06:30:33.481003 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.481129 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:30:33.482117 13989 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:33.482278 13989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:33.482431 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.482012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43792)
0619 06:30:33.482153 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.482266 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260619 06:30:33.482661 13989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
W20260619 06:30:33.482683 13982 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44663: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44663: connect: Connection refused (error 111)
I20260619 06:30:33.485786 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data/instance:
uuid: "dbd9b7613d664a399a03f0b2170776c8"
format_stamp: "Formatted at 2026-06-19 06:30:33 on dist-test-slave-4t1f"
server_key: "448c462b68c44684e58e2c99ff1e1f9e"
server_key_iv: "2da47a27966d01263c8d9340053f8dfe"
server_key_version: "encryptionkey@0"
I20260619 06:30:33.486621 13989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/wal/instance:
uuid: "dbd9b7613d664a399a03f0b2170776c8"
format_stamp: "Formatted at 2026-06-19 06:30:33 on dist-test-slave-4t1f"
server_key: "448c462b68c44684e58e2c99ff1e1f9e"
server_key_iv: "2da47a27966d01263c8d9340053f8dfe"
server_key_version: "encryptionkey@0"
I20260619 06:30:33.491683 13989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:30:33.495137 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:33.496435 13989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:30:33.496615 13989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/wal
uuid: "dbd9b7613d664a399a03f0b2170776c8"
format_stamp: "Formatted at 2026-06-19 06:30:33 on dist-test-slave-4t1f"
server_key: "448c462b68c44684e58e2c99ff1e1f9e"
server_key_iv: "2da47a27966d01263c8d9340053f8dfe"
server_key_version: "encryptionkey@0"
I20260619 06:30:33.496831 13989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:33.515033 13989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:33.518870 13989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:33.519158 13989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:33.537029 13989 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:44663
I20260619 06:30:33.537117 14055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:44663 every 8 connection(s)
I20260619 06:30:33.538352 13989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
I20260619 06:30:33.541791 14056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:33.544108 13908 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 3
I20260619 06:30:33.547899 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 13989
I20260619 06:30:33.548089 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781850433974487-389-0/minicluster-data/master-2/wal/instance
I20260619 06:30:33.548540 389 external_mini_cluster.cc:1428] Setting key 6ea66c0142ee6caecfa406b3d53435b4
I20260619 06:30:33.551483 14056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:33.574786 14060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.550187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.0.97.124:44663)
0619 06:30:33.550946 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.550951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.550973 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:30:33.551019 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.551024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.551083 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.551202 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.552833 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.553814 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.555871 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.556616 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.556735 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.562360 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.562367 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.562372 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.562410 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.570361 (+ 7951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.571322 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.571329 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.571332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.571468 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.571828 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.571833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.571836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.572316 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.572654 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.573027 (+ 373us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.573547 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":600,"thread_start_us":437,"threads_started":1}
I20260619 06:30:33.576370 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.553989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:41493 (local address 127.0.0.1:48804)
0619 06:30:33.554794 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.554840 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.555027 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.555529 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.555534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.556234 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.556557 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.556566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.558087 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.558091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.559515 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.559528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.560570 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.560577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.560757 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.561555 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.561582 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.567602 (+ 6020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.573064 (+ 5462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.573103 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.573113 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.573796 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.574254 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.574263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.574268 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.574461 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.575267 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.575275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.575464 (+ 189us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.575656 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.575966 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"mutex_wait_us":1,"thread_start_us":470,"threads_started":1}
I20260619 06:30:33.577167 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.554132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.0.97.126:41493)
0619 06:30:33.554300 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.554305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.554919 (+ 614us) server_negotiation.cc:407] Connection header received
0619 06:30:33.555163 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.555169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.555238 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.555308 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.556731 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.557868 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.559667 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.560392 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.560527 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.569382 (+ 8855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.569389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.569394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.569444 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.572899 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.573946 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.573952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.573956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.574117 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.574597 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.574604 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.574607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.575033 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.575193 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.576669 (+ 1476us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.576848 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:30:33.578572 14056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:30:33.581264 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.544760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43806)
0619 06:30:33.545019 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.545041 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.545203 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.549138 (+ 3935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.549146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.549688 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.549902 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.549914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.552328 (+ 2414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.552334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.553711 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.553724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.555180 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.555188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.555374 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.556077 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.556100 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.558951 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.568159 (+ 9208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.568166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.568170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.568689 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.569227 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.569234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.569239 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.579567 (+ 10328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.580188 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.580194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.580382 (+ 188us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.580512 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.580769 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260619 06:30:33.582325 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.545629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.0.97.124:44663)
0619 06:30:33.546427 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.546441 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.546494 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:30:33.548177 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.548207 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.548697 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.549141 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.550128 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.552123 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.553868 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.554971 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.555190 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.559257 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.559267 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.559284 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.559349 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.567981 (+ 8632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.568849 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.568857 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.568867 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.569107 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.579699 (+ 10592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.579706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.579709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.580049 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.581584 (+ 1535us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.581700 (+ 116us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.581895 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":348,"threads_started":1}
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:30:33.588076 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: Bootstrap starting.
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:33.591712 13908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:33.594159 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.579290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57940 (local address 127.0.97.125:41299)
0619 06:30:33.579481 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.579486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.579567 (+ 81us) server_negotiation.cc:407] Connection header received
0619 06:30:33.579747 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.579753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.579833 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.579906 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.580890 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.581972 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.583604 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.584033 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.584104 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.587573 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.587577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.587581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.587610 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.590873 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.591716 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.591723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.591728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.591897 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.592377 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.592385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.592389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.592737 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.592856 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.593494 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.593728 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:30:33.594959 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.579229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:41299 (local address 127.0.0.1:57940)
0619 06:30:33.579456 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.579472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.579602 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.580089 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.580096 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.580543 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.580701 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.580712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.582177 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.582187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.583445 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.583462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.584278 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.584281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.584406 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.584980 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.585003 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.587365 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.591037 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.591044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.591047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.591575 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.592066 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.592073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.592078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.592242 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.592960 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.592967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.593171 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.594333 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.594618 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260619 06:30:33.596235 14056 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:30:33.598308 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.578486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48806 (local address 127.0.97.126:41493)
0619 06:30:33.578641 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.578646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.578663 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:30:33.578709 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.578713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.578778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.578873 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.580955 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.582771 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.584958 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.585683 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.585810 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.591137 (+ 5327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.591145 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.591151 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.591188 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.594866 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.595630 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.595637 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.595640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.595768 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.596269 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.596274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.596277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.596657 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.596893 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.597952 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.598102 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:30:33.605149 13908 log.cc:824] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:30:33.611582 13908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: No bootstrap required, opened a new log
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:30:33.617281 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.599848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.0.97.125:41299)
0619 06:30:33.600013 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.600018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.600158 (+ 140us) server_negotiation.cc:407] Connection header received
0619 06:30:33.600303 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.600308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.600424 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.600516 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.601894 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.602635 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.603877 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.604327 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.604476 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.610021 (+ 5545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.610030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.610037 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.610077 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.613859 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.614684 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.614690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.614693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.614838 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.615329 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.615337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.615341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.615765 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.616019 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.616657 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.616902 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:30:33.622826 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:30:33.626066 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.597095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.0.97.124:44663)
0619 06:30:33.605219 (+ 8124us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.605223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.605238 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:30:33.605314 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.605318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.605386 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.605498 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.606373 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.607505 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.609332 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.609984 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.610086 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.614495 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.614501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.614505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.614537 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.618686 (+ 4149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.619882 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.619887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.619890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.620013 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.620745 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.620749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.620751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.621220 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.621367 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.625271 (+ 3904us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.625575 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8002}
I20260619 06:30:33.625947 13908 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.626391 13908 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:33.626519 13908 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad6ba9c439a24b469261a608dd382724, State: Initialized, Role: FOLLOWER
I20260619 06:30:33.626969 14061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.596875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43822)
0619 06:30:33.597172 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.597192 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.597285 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.605512 (+ 8227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.605516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.606007 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.606208 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.606221 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.607657 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.607663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.609173 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.609187 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.610301 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.610307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.610478 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.611243 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.611267 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.614307 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.619307 (+ 5000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.619311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.619314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.619766 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.620142 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.620148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.620151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.620616 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.621414 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.621418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.621572 (+ 154us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.622246 (+ 674us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.622532 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260619 06:30:33.627341 13908 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.628436 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.628898 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:30:33.629885 13908 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:33.633175 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8: Bootstrap starting.
I20260619 06:30:33.638232 14056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:33.639794 14056 log.cc:824] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:33.656509 14056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8: No bootstrap required, opened a new log
I20260619 06:30:33.672077 14056 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.672564 14056 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:33.672704 14056 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd9b7613d664a399a03f0b2170776c8, State: Initialized, Role: FOLLOWER
I20260619 06:30:33.673635 14056 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.684713 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.685186 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:30:33.689785 14056 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:33.700467 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.667661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.0.97.126:41493)
0619 06:30:33.667885 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.667893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.675287 (+ 7394us) server_negotiation.cc:407] Connection header received
0619 06:30:33.675784 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.675792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.675907 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.675992 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.678939 (+ 2947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.680250 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.682627 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.683379 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.683478 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.691585 (+ 8107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.691594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.691602 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.691654 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.696225 (+ 4571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.697330 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.697339 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.697345 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.697547 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.698098 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.698106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.698112 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.698598 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.698905 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.699544 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.699860 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:30:33.713644 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.683642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.0.97.124:44663)
0619 06:30:33.683806 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.683812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.685824 (+ 2012us) server_negotiation.cc:407] Connection header received
0619 06:30:33.686092 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.686100 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.686211 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.686293 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.688303 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.689351 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.691392 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.692274 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.692387 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.702014 (+ 9627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.702024 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.702033 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.702087 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.706619 (+ 4532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.710427 (+ 3808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.710435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.710441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.710631 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.711172 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.711180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.711185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.711644 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.712047 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.712793 (+ 746us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.713125 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:30:33.723941 13987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.667208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57962 (local address 127.0.97.125:41299)
0619 06:30:33.667436 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.667445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.682733 (+ 15288us) server_negotiation.cc:407] Connection header received
0619 06:30:33.685159 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.685165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.685256 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.685350 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.688026 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.689193 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.699619 (+ 10426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.700529 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.700663 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.715238 (+ 14575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.715244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.715251 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.715292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.719734 (+ 4442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.720789 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.720810 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.720816 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.721008 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.721616 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.721623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.721629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.722146 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.722361 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.723110 (+ 749us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.723422 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:30:33.726158 14085 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:30:33.730224 13841 master_runner.cc:186] Error getting master registration for 127.0.97.125:41299: OK, instance_id { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" instance_seqno: 1781850633054984 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:30:33.739470 13841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 06:30:33.743803 14094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:30:33.743948 14094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:30:33.746587 13989 master_runner.cc:186] Error getting master registration for 127.0.97.125:41299: OK, instance_id { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" instance_seqno: 1781850633054984 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:30:33.755509 13989 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 06:30:33.771045 14010 tablet.cc:2369] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:30:33.783179 13982 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } attempt: 5
Jun 19 06:30:33 dist-test-slave-4t1f krb5kdc[13828](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:30:33.798462 14058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.783913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43844 (local address 127.0.97.124:44663)
0619 06:30:33.784116 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:33.784123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:33.784252 (+ 129us) server_negotiation.cc:407] Connection header received
0619 06:30:33.784463 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:33.784468 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:33.784535 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:33.784651 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:33.785581 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.786636 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.788332 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.789011 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.789166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.792762 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:33.792768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:33.792774 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:33.792818 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:33.795742 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.796421 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.796428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.796431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.796572 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:33.796954 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:33.796960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:33.796964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:33.797292 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:33.797449 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:33.797846 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 06:30:33.798051 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:30:33.798480 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:33.783862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44663 (local address 127.0.0.1:43844)
0619 06:30:33.784131 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:33.784150 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:33.784321 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:33.784677 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:33.784682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:33.785210 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:33.785412 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.785424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.786816 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.786821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.788174 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:33.788191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:33.789352 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:33.789357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:33.789486 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:33.790244 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:33.790267 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:33.792579 (+ 2312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:33.795881 (+ 3302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.795888 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.795892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.796303 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.796685 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:33.796690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:33.796694 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:33.796844 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:33.797411 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:33.797415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:33.797583 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 06:30:33.797752 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 06:30:33.798051 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260619 06:30:33.804224 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: Bootstrap starting.
I20260619 06:30:33.807289 13982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:33.808648 13982 log.cc:824] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:33.811280 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: No bootstrap required, opened a new log
I20260619 06:30:33.819216 13982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.819612 13982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:33.819748 13982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67265de3b4f24403b0ed3b2f83c2a22d, State: Initialized, Role: FOLLOWER
I20260619 06:30:33.820472 13982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.822381 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.822988 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:30:33.823726 13982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:30:33.826021 13862 tablet.cc:2369] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:30:33.842612 13915 master_runner.cc:263] No leader master found from master 67265de3b4f24403b0ed3b2f83c2a22d
I20260619 06:30:33.842787 13915 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 06:30:33.847061 14108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:30:33.847210 14108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:30:33.871711 14077 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:30:33.872032 14077 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.873561 14077 leader_election.cc:288] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad6ba9c439a24b469261a608dd382724 (127.0.97.126:41493), 67265de3b4f24403b0ed3b2f83c2a22d (127.0.97.125:41299)
I20260619 06:30:33.874507 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbd9b7613d664a399a03f0b2170776c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" is_pre_election: true
I20260619 06:30:33.875082 13957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd9b7613d664a399a03f0b2170776c8 in term 0.
I20260619 06:30:33.875685 14007 leader_election.cc:302] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67265de3b4f24403b0ed3b2f83c2a22d, dbd9b7613d664a399a03f0b2170776c8; no voters:
I20260619 06:30:33.876770 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbd9b7613d664a399a03f0b2170776c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6ba9c439a24b469261a608dd382724" is_pre_election: true
I20260619 06:30:33.878419 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbd9b7613d664a399a03f0b2170776c8 in term 0.
I20260619 06:30:33.879488 14077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:30:33.879644 14077 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:30:33.879739 14077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:33.883074 14077 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.884042 13883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbd9b7613d664a399a03f0b2170776c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad6ba9c439a24b469261a608dd382724"
I20260619 06:30:33.884253 13883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:33.887406 13883 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd9b7613d664a399a03f0b2170776c8 in term 1.
I20260619 06:30:33.887799 14006 leader_election.cc:302] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad6ba9c439a24b469261a608dd382724, dbd9b7613d664a399a03f0b2170776c8; no voters:
I20260619 06:30:33.890574 13957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbd9b7613d664a399a03f0b2170776c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67265de3b4f24403b0ed3b2f83c2a22d"
I20260619 06:30:33.890807 13957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:33.894289 13957 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbd9b7613d664a399a03f0b2170776c8 in term 1.
I20260619 06:30:33.894953 14077 leader_election.cc:288] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [CANDIDATE]: Term 1 election: Requested vote from peers ad6ba9c439a24b469261a608dd382724 (127.0.97.126:41493), 67265de3b4f24403b0ed3b2f83c2a22d (127.0.97.125:41299)
I20260619 06:30:33.895162 14077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:30:33.895318 14077 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [term 1 LEADER]: Becoming Leader. State: Replica: dbd9b7613d664a399a03f0b2170776c8, State: Running, Role: LEADER
I20260619 06:30:33.896461 14077 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } }
I20260619 06:30:33.909551 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd9b7613d664a399a03f0b2170776c8. Latest consensus state: current_term: 1 leader_uuid: "dbd9b7613d664a399a03f0b2170776c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.909909 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:33.913829 14112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:30:33.916007 14112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:30:33.924950 13957 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [term 1 FOLLOWER]: Refusing update from remote peer dbd9b7613d664a399a03f0b2170776c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:33.925976 14077 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:33.934415 13883 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [term 1 FOLLOWER]: Refusing update from remote peer dbd9b7613d664a399a03f0b2170776c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:33.935316 14110 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:33.944032 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd9b7613d664a399a03f0b2170776c8. Latest consensus state: current_term: 1 leader_uuid: "dbd9b7613d664a399a03f0b2170776c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.944377 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:30:33.948102 14115 mvcc.cc:205] Tried to move back new op lower bound from 7298460196546048000 to 7298460196462964736. Current Snapshot: MvccSnapshot[applied={T|T < 7298460196546048000}]
I20260619 06:30:33.953521 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbd9b7613d664a399a03f0b2170776c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.953809 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:33.958314 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbd9b7613d664a399a03f0b2170776c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.958567 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:30:33.960541 14112 catalog_manager.cc:1347] Generated new cluster ID: 75ac60e1a841405e8125f902c056914d
I20260619 06:30:33.960660 14112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:30:33.961885 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd9b7613d664a399a03f0b2170776c8. Latest consensus state: current_term: 1 leader_uuid: "dbd9b7613d664a399a03f0b2170776c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.962195 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:30:33.963299 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dbd9b7613d664a399a03f0b2170776c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.963634 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:33.977635 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dbd9b7613d664a399a03f0b2170776c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad6ba9c439a24b469261a608dd382724" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41493 } } peers { permanent_uuid: "67265de3b4f24403b0ed3b2f83c2a22d" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 41299 } } peers { permanent_uuid: "dbd9b7613d664a399a03f0b2170776c8" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44663 } } }
I20260619 06:30:33.977892 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:30:33.981554 14112 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:30:33.983634 14112 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:30:33.996677 14112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd9b7613d664a399a03f0b2170776c8: Generated new TSK 0
I20260619 06:30:33.999390 14112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:30:34.730767 14085 catalog_manager.cc:1259] Loaded cluster ID: 75ac60e1a841405e8125f902c056914d
I20260619 06:30:34.730904 14085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: loading cluster ID for follower catalog manager: success
I20260619 06:30:34.733897 14085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: acquiring CA information for follower catalog manager: success
I20260619 06:30:34.735703 14085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ad6ba9c439a24b469261a608dd382724: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:30:34.849325 14108 catalog_manager.cc:1259] Loaded cluster ID: 75ac60e1a841405e8125f902c056914d
I20260619 06:30:34.849462 14108 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: loading cluster ID for follower catalog manager: success
I20260619 06:30:34.851846 14108 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: acquiring CA information for follower catalog manager: success
I20260619 06:30:34.853840 14108 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67265de3b4f24403b0ed3b2f83c2a22d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:30:35.098223 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13841
I20260619 06:30:35.099829 13903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:35.210824 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13841
I20260619 06:30:35.234265 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13915
I20260619 06:30:35.235406 13977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:35.343107 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13915
I20260619 06:30:35.362970 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 13989
I20260619 06:30:35.364006 14051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:35.471134 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 13989
2026-06-19T06:30:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4623 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:30:35 dist-test-slave-4t1f krb5kdc[14131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:30:35 dist-test-slave-4t1f krb5kdc[14131](info): set up 2 sockets
Jun 19 06:30:35 dist-test-slave-4t1f krb5kdc[14131](info): commencing operation
krb5kdc: starting...
W20260619 06:30:36.569510 389 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:30:36 dist-test-slave-4t1f krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:30:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:30:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:30:36.784799 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41687
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:41687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:37.107966 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:37.108294 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:37.108470 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:37.115585 14144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:37.115733 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:37.115832 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:37.115944 14144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:37.116032 14144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:37.125536 14144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:41687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:41687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:37.126811 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:37.128127 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:37.139936 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:37.140028 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:37.139973 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:37.140693 14144 server_base.cc:1034] running on GCE node
I20260619 06:30:37.141551 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:37.142987 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:37.144220 14144 hybrid_clock.cc:648] HybridClock initialized: now 1781850637144181 us; error 39 us; skew 500 ppm
I20260619 06:30:37.144563 14144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:37 dist-test-slave-4t1f krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:37.151592 14144 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:30:37.153239 14144 webserver.cc:460] Webserver started at http://127.0.97.126:35143/ using document root <none> and password file <none>
I20260619 06:30:37.153884 14144 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:37.153983 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:37.154232 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:30:37.157294 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "b8a7078143c043eca4b4a89eba69b5af"
format_stamp: "Formatted at 2026-06-19 06:30:37 on dist-test-slave-4t1f"
server_key: "75e19bef2554f54f0168f8bd32a05516"
server_key_iv: "17830fc88f9859214c753da8de64f1fb"
server_key_version: "encryptionkey@0"
I20260619 06:30:37.157989 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "b8a7078143c043eca4b4a89eba69b5af"
format_stamp: "Formatted at 2026-06-19 06:30:37 on dist-test-slave-4t1f"
server_key: "75e19bef2554f54f0168f8bd32a05516"
server_key_iv: "17830fc88f9859214c753da8de64f1fb"
server_key_version: "encryptionkey@0"
I20260619 06:30:37.162835 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:30:37.166174 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:37.167491 14144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:37.167649 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "b8a7078143c043eca4b4a89eba69b5af"
format_stamp: "Formatted at 2026-06-19 06:30:37 on dist-test-slave-4t1f"
server_key: "75e19bef2554f54f0168f8bd32a05516"
server_key_iv: "17830fc88f9859214c753da8de64f1fb"
server_key_version: "encryptionkey@0"
I20260619 06:30:37.167821 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:37.181660 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:37.185597 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:37.185891 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:37.204312 14144 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:41687
I20260619 06:30:37.204365 14210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:41687 every 8 connection(s)
I20260619 06:30:37.205852 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:30:37.206384 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 14144
I20260619 06:30:37.206616 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:30:37.207468 389 external_mini_cluster.cc:1428] Setting key 5fcbb1c50f7edf652b42d297188a7f3c
I20260619 06:30:37.210386 14211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:30:37 dist-test-slave-4t1f krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:37.221798 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af: Bootstrap starting.
I20260619 06:30:37.225311 14211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:37.226373 14211 log.cc:824] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:37.229246 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:37.209380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.0.97.126:41687)
0619 06:30:37.210141 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:37.210152 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:37.210189 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:30:37.211884 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:37.211904 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:37.212273 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:37.212614 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:37.214052 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:37.215340 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:37.216639 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:37.217352 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:37.217530 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:37.221618 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:37.221625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:37.221641 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:37.221674 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:37.224658 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:37.225451 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:37.225457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:37.225463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:37.225616 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:37.226009 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:37.226013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:37.226015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:37.226542 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:37.226853 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:37.227331 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 06:30:37.227724 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":343,"threads_started":1}
I20260619 06:30:37.230010 14211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af: No bootstrap required, opened a new log
I20260619 06:30:37.234202 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:30:37.240787 14211 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a7078143c043eca4b4a89eba69b5af" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41687 } }
I20260619 06:30:37.241241 14211 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:37.241326 14211 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8a7078143c043eca4b4a89eba69b5af, State: Initialized, Role: FOLLOWER
I20260619 06:30:37.241828 14211 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a7078143c043eca4b4a89eba69b5af" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41687 } }
I20260619 06:30:37.242038 14211 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:30:37.242156 14211 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:30:37.242286 14211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:37.244520 14211 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a7078143c043eca4b4a89eba69b5af" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41687 } }
I20260619 06:30:37.244956 14211 leader_election.cc:302] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8a7078143c043eca4b4a89eba69b5af; no voters:
I20260619 06:30:37.245563 14211 leader_election.cc:288] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:30:37.245720 14216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:30:37.246760 14216 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [term 1 LEADER]: Becoming Leader. State: Replica: b8a7078143c043eca4b4a89eba69b5af, State: Running, Role: LEADER
I20260619 06:30:37.247246 14216 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a7078143c043eca4b4a89eba69b5af" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41687 } }
I20260619 06:30:37.247560 14211 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:37.250558 14217 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8a7078143c043eca4b4a89eba69b5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a7078143c043eca4b4a89eba69b5af" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41687 } } }
I20260619 06:30:37.250532 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8a7078143c043eca4b4a89eba69b5af. Latest consensus state: current_term: 1 leader_uuid: "b8a7078143c043eca4b4a89eba69b5af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8a7078143c043eca4b4a89eba69b5af" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 41687 } } }
I20260619 06:30:37.251215 14217 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:37.251243 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:37.254375 14225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:30:37.258425 14225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:30:37.266168 14225 catalog_manager.cc:1347] Generated new cluster ID: 7e53d267a1eb4447bc23936537298e91
I20260619 06:30:37.266273 14225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:30:37.298252 14225 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:30:37.299559 14225 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:30:37.308995 14225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8a7078143c043eca4b4a89eba69b5af: Generated new TSK 0
I20260619 06:30:37.309872 14225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:30:37.347122 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 14144
I20260619 06:30:37.348317 14206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:37.442610 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 14144
2026-06-19T06:30:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1970 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:30:37 dist-test-slave-4t1f krb5kdc[14239](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:30:37 dist-test-slave-4t1f krb5kdc[14239](info): set up 2 sockets
Jun 19 06:30:37 dist-test-slave-4t1f krb5kdc[14239](info): commencing operation
krb5kdc: starting...
W20260619 06:30:38.537515 389 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:30:38 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:30:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:30:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:30:38.751792 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35347
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:39.062871 14252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:39.063153 14252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:39.063303 14252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:39.070482 14252 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:39.070580 14252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:39.070626 14252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:39.070700 14252 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:39.070787 14252 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:39.080739 14252 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:39.082093 14252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:39.083370 14252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:39.095271 14257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:39.095333 14258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:39.095213 14260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:39.096333 14252 server_base.cc:1034] running on GCE node
I20260619 06:30:39.097098 14252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:39.098449 14252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:39.099697 14252 hybrid_clock.cc:648] HybridClock initialized: now 1781850639099659 us; error 55 us; skew 500 ppm
I20260619 06:30:39.100064 14252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:39 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:39.103060 14252 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:30:39.104566 14252 webserver.cc:460] Webserver started at http://127.0.97.126:35269/ using document root <none> and password file <none>
I20260619 06:30:39.105316 14252 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:39.105461 14252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:39.105759 14252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:30:39.108613 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "a2d60bb7a14c448fa847919b31503477"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "1f2a100f64d76334e546a5b1182bb0fa"
server_key_iv: "93252a986df8400cfe5ef97ba72bafd0"
server_key_version: "encryptionkey@0"
I20260619 06:30:39.109418 14252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "a2d60bb7a14c448fa847919b31503477"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "1f2a100f64d76334e546a5b1182bb0fa"
server_key_iv: "93252a986df8400cfe5ef97ba72bafd0"
server_key_version: "encryptionkey@0"
I20260619 06:30:39.114357 14252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:30:39.117584 14267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:39.118862 14252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:30:39.119041 14252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "a2d60bb7a14c448fa847919b31503477"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "1f2a100f64d76334e546a5b1182bb0fa"
server_key_iv: "93252a986df8400cfe5ef97ba72bafd0"
server_key_version: "encryptionkey@0"
I20260619 06:30:39.119222 14252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:39.143246 14252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:39.147135 14252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:39.147404 14252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:39.165719 14252 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35347
I20260619 06:30:39.165768 14318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35347 every 8 connection(s)
I20260619 06:30:39.167017 14252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:30:39.170687 14319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:39.171257 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 14252
I20260619 06:30:39.171483 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:30:39.172089 389 external_mini_cluster.cc:1428] Setting key 35003a254efd491ecf6c8f9b32019ad0
I20260619 06:30:39.184774 14319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Bootstrap starting.
Jun 19 06:30:39 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:39.187752 14319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:39.188933 14319 log.cc:824] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:39.191746 14319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: No bootstrap required, opened a new log
I20260619 06:30:39.195019 14322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:39.174133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49874 (local address 127.0.97.126:35347)
0619 06:30:39.174888 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:39.174905 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:39.174956 (+ 51us) server_negotiation.cc:407] Connection header received
0619 06:30:39.177241 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:39.177267 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:39.177772 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:39.178359 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:39.179480 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:39.181586 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:39.182874 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:39.183862 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:39.184028 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:39.188009 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:39.188017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:39.188035 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:39.188076 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:39.191192 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:39.191848 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:39.191857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:39.191865 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:39.192047 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:39.192414 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:39.192420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:39.192423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:39.192889 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:39.193227 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:39.193733 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 06:30:39.194121 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":314,"threads_started":1}
I20260619 06:30:39.199676 14319 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:39.200099 14319 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:39.200204 14319 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d60bb7a14c448fa847919b31503477, State: Initialized, Role: FOLLOWER
I20260619 06:30:39.200722 14319 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:39.200966 14319 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:30:39.201122 14319 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:30:39.201320 14319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:39.203429 14319 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:39.203900 14319 leader_election.cc:302] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d60bb7a14c448fa847919b31503477; no voters:
I20260619 06:30:39.204445 14319 leader_election.cc:288] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:30:39.204581 14324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:30:39.205293 14324 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 LEADER]: Becoming Leader. State: Replica: a2d60bb7a14c448fa847919b31503477, State: Running, Role: LEADER
I20260619 06:30:39.205863 14324 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:39.206466 14319 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:39.208778 14325 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d60bb7a14c448fa847919b31503477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } } }
I20260619 06:30:39.209100 14325 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:39.209529 14326 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d60bb7a14c448fa847919b31503477. Latest consensus state: current_term: 1 leader_uuid: "a2d60bb7a14c448fa847919b31503477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } } }
I20260619 06:30:39.209862 14326 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:39.210551 14331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:30:39.215298 14331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:30:39.224845 14331 catalog_manager.cc:1347] Generated new cluster ID: 8f5552e4b7c64979a8a08a9077a0fb49
I20260619 06:30:39.224953 14331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:30:39.247320 14331 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:30:39.248607 14331 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:30:39.256734 14331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Generated new TSK 0
I20260619 06:30:39.257570 14331 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:30:39.324769 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35347
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:39.681744 14345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:39.682024 14345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:39.682123 14345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:39.689632 14345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:39.689800 14345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:39.690001 14345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:30:39.699735 14345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:39.700898 14345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:39.702240 14345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:39.715484 14350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:39.715683 14351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:39.715955 14353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:39.716308 14345 server_base.cc:1034] running on GCE node
I20260619 06:30:39.716895 14345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:39.717859 14345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:39.719071 14345 hybrid_clock.cc:648] HybridClock initialized: now 1781850639718992 us; error 75 us; skew 500 ppm
I20260619 06:30:39.719367 14345 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:39 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:39.722666 14345 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:30:39.724222 14345 webserver.cc:460] Webserver started at http://127.0.97.65:43555/ using document root <none> and password file <none>
I20260619 06:30:39.724866 14345 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:39.725011 14345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:39.725327 14345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:30:39.728071 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "cdd1de95e6b045808f27aff9698adbd7"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "3538b62c478519a1ee4c70fb5f94ca0c"
server_key_iv: "e4254b95fa0f2a5d41af0b031a86440e"
server_key_version: "encryptionkey@0"
I20260619 06:30:39.728858 14345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "cdd1de95e6b045808f27aff9698adbd7"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "3538b62c478519a1ee4c70fb5f94ca0c"
server_key_iv: "e4254b95fa0f2a5d41af0b031a86440e"
server_key_version: "encryptionkey@0"
I20260619 06:30:39.733728 14345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:30:39.736931 14360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:39.738211 14345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:30:39.738389 14345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "cdd1de95e6b045808f27aff9698adbd7"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "3538b62c478519a1ee4c70fb5f94ca0c"
server_key_iv: "e4254b95fa0f2a5d41af0b031a86440e"
server_key_version: "encryptionkey@0"
I20260619 06:30:39.738579 14345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:39.758993 14345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:39.763096 14345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:39.763374 14345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:39.764304 14345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:39.765618 14345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:39.765692 14345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:39.765856 14345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:39.765916 14345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:39.799635 14345 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:38829
I20260619 06:30:39.799695 14472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:38829 every 8 connection(s)
I20260619 06:30:39.800913 14345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:30:39.809671 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 14345
I20260619 06:30:39.809901 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:30:39.810513 389 external_mini_cluster.cc:1428] Setting key 1f129c066daf338bc4665ad175bee026
Jun 19 06:30:39 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850639, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:39.822579 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:39.803480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:56095 (local address 127.0.97.126:35347)
0619 06:30:39.803906 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:39.803911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:39.805022 (+ 1111us) server_negotiation.cc:407] Connection header received
0619 06:30:39.806874 (+ 1852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:39.806879 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:39.806964 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:39.807085 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:39.808498 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:39.809403 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:39.811373 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:39.811806 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:39.811904 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:39.816233 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:39.816239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:39.816243 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:39.816275 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:39.819009 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:39.819988 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:39.819997 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:39.820002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:39.820164 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:39.820595 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:39.820602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:39.820609 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:39.821009 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:39.821206 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:39.821823 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 06:30:39.822115 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":210,"threads_started":1}
I20260619 06:30:39.823226 14476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:39.804029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:56095)
0619 06:30:39.804855 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:39.804899 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:39.806602 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:39.807315 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:39.807323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:39.807801 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:39.808319 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:39.808335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:39.809589 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:39.809593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:39.811181 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:39.811198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:39.812039 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:39.812046 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:39.812396 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:39.813246 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:39.813278 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:39.815966 (+ 2688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:39.819173 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:39.819184 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:39.819202 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:39.819811 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:39.820279 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:39.820282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:39.820284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:39.820483 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:39.821170 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:39.821175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:39.821453 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 06:30:39.821668 (+ 215us) client_negotiation.cc:240] Negotiation successful
0619 06:30:39.822034 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":413,"threads_started":1}
I20260619 06:30:39.824738 14473 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35347
I20260619 06:30:39.825024 14473 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:39.825706 14473 heartbeater.cc:511] Master 127.0.97.126:35347 requested a full tablet report, sending...
I20260619 06:30:39.827651 14284 ts_manager.cc:194] Registered new tserver with Master: cdd1de95e6b045808f27aff9698adbd7 (127.0.97.65:38829)
I20260619 06:30:39.829018 14284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:56095
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:30:39.860292 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35347
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:40.188122 14479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:40.188438 14479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:40.188611 14479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:40.196502 14479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:40.196635 14479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:40.196803 14479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:30:40.206625 14479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:40.208006 14479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:40.209460 14479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:40.222818 14484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:40.222767 14487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:40.222862 14485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:40.223667 14479 server_base.cc:1034] running on GCE node
I20260619 06:30:40.224246 14479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:40.225173 14479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:40.226424 14479 hybrid_clock.cc:648] HybridClock initialized: now 1781850640226331 us; error 83 us; skew 500 ppm
I20260619 06:30:40.226671 14479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:40 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:40.229698 14479 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:30:40.231041 14479 webserver.cc:460] Webserver started at http://127.0.97.66:41895/ using document root <none> and password file <none>
I20260619 06:30:40.231673 14479 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:40.231796 14479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:40.232087 14479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:30:40.234854 14479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "3051d8db01164019bbadb94f59ceda43"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "33220417311254b547b8406590a0dab8"
server_key_iv: "56b5edcc41ff80ea2f54597cd9f8f7e5"
server_key_version: "encryptionkey@0"
I20260619 06:30:40.235592 14479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "3051d8db01164019bbadb94f59ceda43"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "33220417311254b547b8406590a0dab8"
server_key_iv: "56b5edcc41ff80ea2f54597cd9f8f7e5"
server_key_version: "encryptionkey@0"
I20260619 06:30:40.240927 14479 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:30:40.244779 14494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:40.246050 14479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:40.246217 14479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "3051d8db01164019bbadb94f59ceda43"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "33220417311254b547b8406590a0dab8"
server_key_iv: "56b5edcc41ff80ea2f54597cd9f8f7e5"
server_key_version: "encryptionkey@0"
I20260619 06:30:40.246445 14479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:40.267683 14479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:40.271262 14479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:40.271538 14479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:40.272476 14479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:40.273737 14479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:40.273836 14479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:40.273985 14479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:40.274076 14479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:40.305897 14479 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39175
I20260619 06:30:40.305961 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39175 every 8 connection(s)
I20260619 06:30:40.307163 14479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:30:40.310135 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 14479
I20260619 06:30:40.310334 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:30:40.310956 389 external_mini_cluster.cc:1428] Setting key 19082e3d1b387e9f6d926a4fba8af092
Jun 19 06:30:40 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:40.327633 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.309684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:49111 (local address 127.0.97.126:35347)
0619 06:30:40.309850 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:40.309854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:40.311350 (+ 1496us) server_negotiation.cc:407] Connection header received
0619 06:30:40.313459 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:40.313468 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:40.313561 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:40.313667 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:40.315166 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.315992 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.317730 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.318184 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.318317 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.321681 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:40.321686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:40.321689 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:40.321720 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:40.324563 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:40.325411 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:40.325415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:40.325417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:40.325533 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:40.325942 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:40.325947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:40.325952 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:40.326201 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:40.326358 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:40.327042 (+ 684us) server_negotiation.cc:299] Negotiation successful
0619 06:30:40.327301 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:30:40.328624 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.310340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:49111)
0619 06:30:40.311174 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:40.311216 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:40.313184 (+ 1968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:40.313921 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:40.313930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:40.314419 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:40.314987 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.315005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.316145 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.316149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.317584 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.317597 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.318401 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.318408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.318637 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.319333 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:40.319356 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:40.321475 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:40.324697 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:40.324706 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:40.324721 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:40.325286 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:40.325638 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:40.325644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:40.325648 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:40.325827 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:40.326329 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:40.326336 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:40.326657 (+ 321us) client_negotiation.cc:769] Sending connection context
0619 06:30:40.326942 (+ 285us) client_negotiation.cc:240] Negotiation successful
0619 06:30:40.327437 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":301,"threads_started":1}
I20260619 06:30:40.330027 14607 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35347
I20260619 06:30:40.330322 14607 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:40.330982 14607 heartbeater.cc:511] Master 127.0.97.126:35347 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
I20260619 06:30:40.332350 14284 ts_manager.cc:194] Registered new tserver with Master: 3051d8db01164019bbadb94f59ceda43 (127.0.97.66:39175)
I20260619 06:30:40.333312 14284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:49111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:30:40.359998 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35347
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:40.677390 14612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:40.677635 14612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:40.677726 14612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:40.684629 14612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:40.684726 14612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:40.684831 14612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:30:40.693924 14612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:40.694974 14612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:40.696106 14612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:40.708891 14618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:40.709357 14620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:40.709484 14617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:40.709933 14612 server_base.cc:1034] running on GCE node
I20260619 06:30:40.710454 14612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:40.711311 14612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:40.712576 14612 hybrid_clock.cc:648] HybridClock initialized: now 1781850640712535 us; error 32 us; skew 500 ppm
I20260619 06:30:40.712888 14612 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:40 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:40.715950 14612 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:30:40.717490 14612 webserver.cc:460] Webserver started at http://127.0.97.67:33769/ using document root <none> and password file <none>
I20260619 06:30:40.718111 14612 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:40.718251 14612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:40.718535 14612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:30:40.721333 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "221f10cb7eef44f196876a0f06deb3da"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "e74e15d98a3a1f1032d6fbce31dd7e75"
server_key_iv: "71f13618742f95ae227f0eac0a4f194a"
server_key_version: "encryptionkey@0"
I20260619 06:30:40.722170 14612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "221f10cb7eef44f196876a0f06deb3da"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "e74e15d98a3a1f1032d6fbce31dd7e75"
server_key_iv: "71f13618742f95ae227f0eac0a4f194a"
server_key_version: "encryptionkey@0"
I20260619 06:30:40.727006 14612 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:30:40.730286 14627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:40.731581 14612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:40.731760 14612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "221f10cb7eef44f196876a0f06deb3da"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "e74e15d98a3a1f1032d6fbce31dd7e75"
server_key_iv: "71f13618742f95ae227f0eac0a4f194a"
server_key_version: "encryptionkey@0"
I20260619 06:30:40.731943 14612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:40.744145 14612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:40.747715 14612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:40.747941 14612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:40.748726 14612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:40.749955 14612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:40.750025 14612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:40.750098 14612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:40.750149 14612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:40.782943 14612 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:36779
I20260619 06:30:40.782991 14739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:36779 every 8 connection(s)
I20260619 06:30:40.784192 14612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:30:40.790115 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 14612
I20260619 06:30:40.790307 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:30:40.790831 389 external_mini_cluster.cc:1428] Setting key cd643ff3a010353a18fcd1e41bf7545f
Jun 19 06:30:40 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:40.804697 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.786625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45811 (local address 127.0.97.126:35347)
0619 06:30:40.786838 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:40.786844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:40.787885 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 06:30:40.789626 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:40.789631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:40.789774 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:40.789885 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:40.791861 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.792901 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.794668 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.795051 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.795153 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.798432 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:40.798437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:40.798440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:40.798470 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:40.801694 (+ 3224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:40.802424 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:40.802430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:40.802434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:40.802581 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:40.803010 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:40.803015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:40.803019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:40.803361 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:40.803496 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:40.804123 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 06:30:40.804348 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:30:40.805851 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.787060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:45811)
0619 06:30:40.787734 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:40.787776 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:40.789380 (+ 1604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:40.790173 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:40.790187 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:40.790902 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:40.791682 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.791706 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.793044 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.793048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.794528 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.794539 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.795252 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.795256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.795433 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.796041 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:40.796062 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:40.798252 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:40.801833 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:40.801851 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:40.801862 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:40.802292 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:40.802689 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:40.802692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:40.802695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:40.802854 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:40.803474 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:40.803480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:40.803758 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 06:30:40.803963 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 06:30:40.804475 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":274,"threads_started":1}
I20260619 06:30:40.807132 14740 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35347
I20260619 06:30:40.807474 14740 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:40.808203 14740 heartbeater.cc:511] Master 127.0.97.126:35347 requested a full tablet report, sending...
I20260619 06:30:40.809679 14284 ts_manager.cc:194] Registered new tserver with Master: 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779)
I20260619 06:30:40.810461 14284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:45811
I20260619 06:30:40.818557 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:30:40.831791 14473 heartbeater.cc:503] Master 127.0.97.126:35347 was elected leader, sending a full tablet report...
I20260619 06:30:40.835843 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.822974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49876 (local address 127.0.97.126:35347)
0619 06:30:40.823261 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:40.823270 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:40.823581 (+ 311us) server_negotiation.cc:407] Connection header received
0619 06:30:40.823730 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:40.823733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:40.823791 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:40.823880 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:40.825324 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.826157 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.827415 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.827862 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.827974 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.830353 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:40.830358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:40.830361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:40.830390 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:40.833131 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:40.833850 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:40.833858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:40.833866 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:40.834035 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:40.834404 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:40.834410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:40.834413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:40.834706 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:40.835003 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:40.835316 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 06:30:40.835529 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260619 06:30:40.839422 14284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:30:40.846235 14284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:30:40.873294 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.859271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39175 (local address 127.0.0.1:49062)
0619 06:30:40.860632 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:40.860706 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:40.860966 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:40.865783 (+ 4817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:40.865789 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:40.865819 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:40.866391 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.866409 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.868791 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.868797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.871076 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.871085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.872337 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.872343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.872524 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.872575 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 06:30:40.872724 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 06:30:40.872914 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"thread_start_us":5653,"threads_started":1}
I20260619 06:30:40.873272 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.859313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49062 (local address 127.0.97.66:39175)
0619 06:30:40.864867 (+ 5554us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:40.864877 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:40.864906 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:30:40.865008 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:40.865021 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:40.865315 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:40.866642 (+ 1327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:40.866942 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.868579 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.871219 (+ 2640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.872151 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.872305 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.872521 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:40.872762 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 06:30:40.872925 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5385,"thread_start_us":5239,"threads_started":1}
I20260619 06:30:40.874264 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.861107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44176)
0619 06:30:40.865222 (+ 4115us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:40.865238 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:40.865428 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:40.866441 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:40.866448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:40.866483 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:40.866927 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.866941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.868366 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.868371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.870820 (+ 2449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.870834 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.873264 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.873272 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.873491 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.873531 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:30:40.873623 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 06:30:40.873760 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3972,"mutex_wait_us":89,"thread_start_us":2858,"threads_started":1}
I20260619 06:30:40.874280 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.860185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.0.97.67:36779)
0619 06:30:40.863273 (+ 3088us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:40.863287 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:40.864366 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 06:30:40.864684 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:40.864696 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:40.864977 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:40.865227 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:40.865931 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.867299 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.870472 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.872201 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.872359 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.872640 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:40.873300 (+ 660us) server_negotiation.cc:299] Negotiation successful
0619 06:30:40.873853 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2781,"thread_start_us":1582,"threads_started":1}
I20260619 06:30:40.874281 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.861630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44176 (local address 127.0.97.65:38829)
0619 06:30:40.863972 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:40.863994 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:40.865324 (+ 1330us) server_negotiation.cc:407] Connection header received
0619 06:30:40.865654 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:40.865665 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:40.865910 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:40.866129 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:40.867117 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.868231 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.870997 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.872693 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.872835 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.873105 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:40.873799 (+ 694us) server_negotiation.cc:299] Negotiation successful
0619 06:30:40.873970 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":1967,"threads_started":1}
I20260619 06:30:40.875294 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:40.859781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36779 (local address 127.0.0.1:51716)
0619 06:30:40.864226 (+ 4445us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:40.864243 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:40.864418 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:40.865222 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:40.865230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:40.865273 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:40.865727 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.865739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.867463 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.867470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.870243 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:40.870266 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:40.872794 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:40.872802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:40.873005 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:40.873087 (+ 82us) client_negotiation.cc:769] Sending connection context
0619 06:30:40.874793 (+ 1706us) client_negotiation.cc:240] Negotiation successful
0619 06:30:40.874918 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4337,"mutex_wait_us":74,"thread_start_us":373,"threads_started":1}
I20260619 06:30:40.877249 14542 tablet_service.cc:1467] Processing CreateTablet for tablet c888e1d9be0b4513b558767e6f8b4989 (DEFAULT_TABLE table=test-table [id=d6a97eb988ac439f846f4715b487533a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:40.878422 14542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c888e1d9be0b4513b558767e6f8b4989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:40.878299 14408 tablet_service.cc:1467] Processing CreateTablet for tablet c888e1d9be0b4513b558767e6f8b4989 (DEFAULT_TABLE table=test-table [id=d6a97eb988ac439f846f4715b487533a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:40.879106 14675 tablet_service.cc:1467] Processing CreateTablet for tablet c888e1d9be0b4513b558767e6f8b4989 (DEFAULT_TABLE table=test-table [id=d6a97eb988ac439f846f4715b487533a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:40.879841 14408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c888e1d9be0b4513b558767e6f8b4989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:40.880574 14675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c888e1d9be0b4513b558767e6f8b4989. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:40.889920 14758 tablet_bootstrap.cc:492] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Bootstrap starting.
I20260619 06:30:40.892949 14758 tablet_bootstrap.cc:654] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:40.893934 14758 log.cc:824] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:40.895975 14760 tablet_bootstrap.cc:492] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Bootstrap starting.
I20260619 06:30:40.897642 14761 tablet_bootstrap.cc:492] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Bootstrap starting.
I20260619 06:30:40.897857 14758 tablet_bootstrap.cc:492] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: No bootstrap required, opened a new log
I20260619 06:30:40.898123 14758 ts_tablet_manager.cc:1397] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260619 06:30:40.900352 14760 tablet_bootstrap.cc:654] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:40.901360 14761 tablet_bootstrap.cc:654] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:40.901559 14760 log.cc:824] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:40.902774 14761 log.cc:824] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:40.905289 14760 tablet_bootstrap.cc:492] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: No bootstrap required, opened a new log
I20260619 06:30:40.905546 14760 ts_tablet_manager.cc:1397] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260619 06:30:40.905668 14761 tablet_bootstrap.cc:492] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: No bootstrap required, opened a new log
I20260619 06:30:40.905941 14761 ts_tablet_manager.cc:1397] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260619 06:30:40.908601 14758 raft_consensus.cc:348] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:40.908993 14758 raft_consensus.cc:374] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:40.909093 14758 raft_consensus.cc:729] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3051d8db01164019bbadb94f59ceda43, State: Initialized, Role: FOLLOWER
I20260619 06:30:40.909786 14758 consensus_queue.cc:260] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:40.910985 14607 heartbeater.cc:503] Master 127.0.97.126:35347 was elected leader, sending a full tablet report...
I20260619 06:30:40.911513 14758 ts_tablet_manager.cc:1428] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Time spent starting tablet: real 0.013s user 0.006s sys 0.007s
I20260619 06:30:40.915515 14760 raft_consensus.cc:348] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:40.915899 14760 raft_consensus.cc:374] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:40.915992 14760 raft_consensus.cc:729] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd1de95e6b045808f27aff9698adbd7, State: Initialized, Role: FOLLOWER
I20260619 06:30:40.916000 14761 raft_consensus.cc:348] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:40.916503 14761 raft_consensus.cc:374] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:40.916641 14761 raft_consensus.cc:729] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f10cb7eef44f196876a0f06deb3da, State: Initialized, Role: FOLLOWER
I20260619 06:30:40.916530 14760 consensus_queue.cc:260] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:40.917495 14761 consensus_queue.cc:260] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:40.918272 14760 ts_tablet_manager.cc:1428] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260619 06:30:40.918568 14740 heartbeater.cc:503] Master 127.0.97.126:35347 was elected leader, sending a full tablet report...
I20260619 06:30:40.919310 14761 ts_tablet_manager.cc:1428] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
W20260619 06:30:41.035816 14741 tablet.cc:2369] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:30:41.053354 14474 tablet.cc:2369] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:30:41.059131 14608 tablet.cc:2369] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:30:41.204604 14765 raft_consensus.cc:482] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:30:41.204954 14765 raft_consensus.cc:504] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:41.206833 14765 leader_election.cc:288] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3051d8db01164019bbadb94f59ceda43 (127.0.97.66:39175), 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779)
I20260619 06:30:41.213944 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.207235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:43363 (local address 127.0.97.66:39175)
0619 06:30:41.207466 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:41.207472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:41.208389 (+ 917us) server_negotiation.cc:407] Connection header received
0619 06:30:41.208591 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:41.208600 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:41.208688 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:41.208772 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:41.209510 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.210217 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.212067 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.212895 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.212991 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.213110 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:41.213525 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 06:30:41.213685 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:30:41.214007 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.207407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42305 (local address 127.0.97.67:36779)
0619 06:30:41.207617 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:41.207622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:41.208483 (+ 861us) server_negotiation.cc:407] Connection header received
0619 06:30:41.208671 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:41.208676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:41.208753 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:41.208869 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:41.209587 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.210298 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.212143 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.212998 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.213167 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.213266 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:41.213617 (+ 351us) server_negotiation.cc:299] Negotiation successful
0619 06:30:41.213768 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:30:41.214077 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.207341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39175 (local address 127.0.97.65:43363)
0619 06:30:41.208267 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:41.208290 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:41.208413 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:41.208821 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:41.208827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:41.208862 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:41.209361 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:41.209371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.210355 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.210363 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:41.211902 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:41.211917 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.213140 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.213146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:41.213328 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.213368 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:30:41.213442 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 06:30:41.213617 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":2,"spinlock_wait_cycles":768,"thread_start_us":628,"threads_started":1}
I20260619 06:30:41.214118 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.207384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36779 (local address 127.0.97.65:42305)
0619 06:30:41.208358 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:41.208380 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:41.208533 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:41.208903 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:41.208907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:41.208935 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:41.209395 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:41.209418 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.210433 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.210436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:41.211948 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:41.211960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.213237 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.213241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:41.213377 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.213399 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:30:41.213563 (+ 164us) client_negotiation.cc:240] Negotiation successful
0619 06:30:41.213706 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":85,"thread_start_us":526,"threads_started":1}
I20260619 06:30:41.215169 14695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c888e1d9be0b4513b558767e6f8b4989" candidate_uuid: "cdd1de95e6b045808f27aff9698adbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f10cb7eef44f196876a0f06deb3da" is_pre_election: true
I20260619 06:30:41.215169 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c888e1d9be0b4513b558767e6f8b4989" candidate_uuid: "cdd1de95e6b045808f27aff9698adbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3051d8db01164019bbadb94f59ceda43" is_pre_election: true
I20260619 06:30:41.215793 14562 raft_consensus.cc:2454] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd1de95e6b045808f27aff9698adbd7 in term 0.
I20260619 06:30:41.215780 14695 raft_consensus.cc:2454] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdd1de95e6b045808f27aff9698adbd7 in term 0.
I20260619 06:30:41.216512 14363 leader_election.cc:302] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 221f10cb7eef44f196876a0f06deb3da, 3051d8db01164019bbadb94f59ceda43, cdd1de95e6b045808f27aff9698adbd7; no voters:
I20260619 06:30:41.216971 14765 raft_consensus.cc:2781] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:30:41.217151 14765 raft_consensus.cc:482] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:30:41.217284 14765 raft_consensus.cc:3037] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:41.219398 14765 raft_consensus.cc:504] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:41.219941 14765 leader_election.cc:288] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [CANDIDATE]: Term 1 election: Requested vote from peers 3051d8db01164019bbadb94f59ceda43 (127.0.97.66:39175), 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779)
I20260619 06:30:41.220410 14695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c888e1d9be0b4513b558767e6f8b4989" candidate_uuid: "cdd1de95e6b045808f27aff9698adbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f10cb7eef44f196876a0f06deb3da"
I20260619 06:30:41.220431 14562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c888e1d9be0b4513b558767e6f8b4989" candidate_uuid: "cdd1de95e6b045808f27aff9698adbd7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3051d8db01164019bbadb94f59ceda43"
I20260619 06:30:41.220628 14695 raft_consensus.cc:3037] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:41.220618 14562 raft_consensus.cc:3037] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:41.223780 14562 raft_consensus.cc:2454] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd1de95e6b045808f27aff9698adbd7 in term 1.
I20260619 06:30:41.223780 14695 raft_consensus.cc:2454] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdd1de95e6b045808f27aff9698adbd7 in term 1.
I20260619 06:30:41.224149 14361 leader_election.cc:302] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 221f10cb7eef44f196876a0f06deb3da, cdd1de95e6b045808f27aff9698adbd7; no voters:
I20260619 06:30:41.224453 14765 raft_consensus.cc:2781] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:30:41.224891 14765 raft_consensus.cc:686] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 1 LEADER]: Becoming Leader. State: Replica: cdd1de95e6b045808f27aff9698adbd7, State: Running, Role: LEADER
I20260619 06:30:41.225353 14765 consensus_queue.cc:237] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } }
I20260619 06:30:41.229743 14283 catalog_manager.cc:5495] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 reported cstate change: term changed from 0 to 1, leader changed from <none> to cdd1de95e6b045808f27aff9698adbd7 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "cdd1de95e6b045808f27aff9698adbd7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } health_report { overall_health: HEALTHY } } }
I20260619 06:30:41.323562 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.318459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44182 (local address 127.0.97.65:38829)
0619 06:30:41.318637 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:41.318642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:41.318685 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:30:41.318877 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:41.318881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:41.318969 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:41.319082 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:30:41.319396 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.320083 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.321645 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.322056 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.322152 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.322579 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:30:41.323029 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:30:41.323133 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:41.323238 (+ 105us) server_negotiation.cc:299] Negotiation successful
0619 06:30:41.323297 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:30:41.331230 14695 raft_consensus.cc:1261] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Refusing update from remote peer cdd1de95e6b045808f27aff9698adbd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:41.331228 14562 raft_consensus.cc:1261] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 1 FOLLOWER]: Refusing update from remote peer cdd1de95e6b045808f27aff9698adbd7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:41.332166 14769 consensus_queue.cc:1035] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:41.332464 14765 consensus_queue.cc:1035] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:41.348140 14772 mvcc.cc:205] Tried to move back new op lower bound from 7298460226883518464 to 7298460226462441472. Current Snapshot: MvccSnapshot[applied={T|T < 7298460226883518464}]
I20260619 06:30:41.361521 14283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:30:41.361826 14283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:30:41.365155 14283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Sending DeleteTablet for 3 replicas of tablet c888e1d9be0b4513b558767e6f8b4989
I20260619 06:30:41.366307 14675 tablet_service.cc:1514] Processing DeleteTablet for tablet c888e1d9be0b4513b558767e6f8b4989 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:41 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51716
I20260619 06:30:41.366328 14542 tablet_service.cc:1514] Processing DeleteTablet for tablet c888e1d9be0b4513b558767e6f8b4989 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:41 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49062
I20260619 06:30:41.366386 14408 tablet_service.cc:1514] Processing DeleteTablet for tablet c888e1d9be0b4513b558767e6f8b4989 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:41 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44176
I20260619 06:30:41.367596 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 14345
I20260619 06:30:41.381819 14783 tablet_replica.cc:330] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: stopping tablet replica
I20260619 06:30:41.382375 14783 raft_consensus.cc:2229] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:30:41.382958 14783 raft_consensus.cc:2258] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:41.383800 14782 tablet_replica.cc:330] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: stopping tablet replica
I20260619 06:30:41.384212 14782 raft_consensus.cc:2229] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:30:41.384831 14784 tablet_replica.cc:330] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: stopping tablet replica
I20260619 06:30:41.384876 14782 raft_consensus.cc:2258] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:41.385255 14784 raft_consensus.cc:2229] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:30:41.385728 14784 raft_consensus.cc:2258] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:41.387389 14782 ts_tablet_manager.cc:1905] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:41.387632 14784 ts_tablet_manager.cc:1905] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:41.394629 14784 ts_tablet_manager.cc:1918] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:41.394771 14782 ts_tablet_manager.cc:1918] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:41.394804 14784 log.cc:1196] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/c888e1d9be0b4513b558767e6f8b4989
I20260619 06:30:41.394934 14782 log.cc:1196] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/c888e1d9be0b4513b558767e6f8b4989
I20260619 06:30:41.395354 14784 ts_tablet_manager.cc:1939] T c888e1d9be0b4513b558767e6f8b4989 P 3051d8db01164019bbadb94f59ceda43: Deleting consensus metadata
I20260619 06:30:41.395459 14782 ts_tablet_manager.cc:1939] T c888e1d9be0b4513b558767e6f8b4989 P cdd1de95e6b045808f27aff9698adbd7: Deleting consensus metadata
Jun 19 06:30:41 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:30:41.397131 14268 catalog_manager.cc:4841] TS cdd1de95e6b045808f27aff9698adbd7 (127.0.97.65:38829): tablet c888e1d9be0b4513b558767e6f8b4989 (table test-table [id=d6a97eb988ac439f846f4715b487533a]) successfully deleted
I20260619 06:30:41.397138 14270 catalog_manager.cc:4841] TS 3051d8db01164019bbadb94f59ceda43 (127.0.97.66:39175): tablet c888e1d9be0b4513b558767e6f8b4989 (table test-table [id=d6a97eb988ac439f846f4715b487533a]) successfully deleted
I20260619 06:30:41.398478 14783 ts_tablet_manager.cc:1905] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:41.404713 14783 ts_tablet_manager.cc:1918] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:41.404876 14783 log.cc:1196] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/c888e1d9be0b4513b558767e6f8b4989
I20260619 06:30:41.405416 14783 ts_tablet_manager.cc:1939] T c888e1d9be0b4513b558767e6f8b4989 P 221f10cb7eef44f196876a0f06deb3da: Deleting consensus metadata
I20260619 06:30:41.406329 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.372796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.0.97.65:38829)
0619 06:30:41.373027 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:41.373034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:41.384188 (+ 11154us) server_negotiation.cc:407] Connection header received
0619 06:30:41.384412 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:41.384419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:41.384503 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:41.384638 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:41.387073 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.388229 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.390623 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.391358 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.391473 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.398612 (+ 7139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:41.398627 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:41.398644 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:41.398684 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:41.402764 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:41.403692 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:41.403702 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:41.403706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:41.403888 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:41.404385 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:41.404390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:41.404393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:41.404874 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:41.405267 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:41.405762 (+ 495us) server_negotiation.cc:299] Negotiation successful
0619 06:30:41.405995 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260619 06:30:41.407127 14268 catalog_manager.cc:4841] TS 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779): tablet c888e1d9be0b4513b558767e6f8b4989 (table test-table [id=d6a97eb988ac439f846f4715b487533a]) successfully deleted
I20260619 06:30:41.407263 14468 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:41.505295 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 14345
I20260619 06:30:41.523676 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 14479
Jun 19 06:30:41 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:30:41.538439 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.524443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.0.97.66:39175)
0619 06:30:41.524641 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:41.524646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:41.524663 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:30:41.524753 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:41.524757 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:41.524819 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:41.524902 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:41.526216 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.526925 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.528221 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.528640 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.528729 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.533044 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:41.533057 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:41.533109 (+ 52us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:41.533153 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:41.535916 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:41.536615 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:41.536623 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:41.536625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:41.536756 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:41.537111 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:41.537115 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:41.537118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:41.537498 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:41.537722 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:41.538017 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:30:41.538164 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:30:41.539134 14602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:41.627276 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 14479
I20260619 06:30:41.645556 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 14612
Jun 19 06:30:41 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:30:41.659479 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:41.646378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.0.97.67:36779)
0619 06:30:41.646559 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:41.646564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:41.646583 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:30:41.646675 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:41.646680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:41.646742 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:41.646821 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:41.648020 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.648712 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.649906 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:41.650282 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:41.650378 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:41.654456 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:41.654468 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:41.654483 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:41.654516 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:41.657007 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:41.657682 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:41.657689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:41.657691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:41.657817 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:41.658158 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:41.658160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:41.658162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:41.658504 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:41.658703 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:41.658956 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:30:41.659107 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:30:41.660147 14735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:41.747998 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 14612
I20260619 06:30:41.764045 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 14252
I20260619 06:30:41.765233 14314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:41.872601 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 14252
I20260619 06:30:41.893918 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35347
--webserver_interface=127.0.97.126
--webserver_port=35269
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:42.216702 14790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:42.216981 14790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:42.217180 14790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:42.224406 14790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:42.224500 14790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:42.224601 14790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:42.224696 14790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:42.224779 14790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:42.234515 14790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:42.235806 14790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:42.237107 14790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:42.259510 14799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:42.259523 14796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:42.259495 14797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:42.260691 14790 server_base.cc:1034] running on GCE node
I20260619 06:30:42.261672 14790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:42.263213 14790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:42.264442 14790 hybrid_clock.cc:648] HybridClock initialized: now 1781850642264378 us; error 51 us; skew 500 ppm
I20260619 06:30:42.264835 14790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:42 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:42.273600 14790 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:30:42.275380 14790 webserver.cc:460] Webserver started at http://127.0.97.126:35269/ using document root <none> and password file <none>
I20260619 06:30:42.276152 14790 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:42.276315 14790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:42.282349 14790 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:30:42.285470 14806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:42.286847 14790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:30:42.287055 14790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "a2d60bb7a14c448fa847919b31503477"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "1f2a100f64d76334e546a5b1182bb0fa"
server_key_iv: "93252a986df8400cfe5ef97ba72bafd0"
server_key_version: "encryptionkey@0"
I20260619 06:30:42.288093 14790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:42.304715 14790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:42.309175 14790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:42.309469 14790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:42.328593 14790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35347
I20260619 06:30:42.328617 14857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35347 every 8 connection(s)
I20260619 06:30:42.330142 14790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:30:42.334636 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 14790
I20260619 06:30:42.335568 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:38829
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:35347
--webserver_port=43555
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:30:42.338184 14858 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:30:42.343374 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Bootstrap starting.
I20260619 06:30:42.362691 14858 log.cc:824] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:42.374688 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:30:42.375208 14858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Bootstrap complete.
I20260619 06:30:42.384652 14858 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:42.385537 14858 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d60bb7a14c448fa847919b31503477, State: Initialized, Role: FOLLOWER
I20260619 06:30:42.386196 14858 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:42.386474 14858 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:30:42.386596 14858 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:30:42.386780 14858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:30:42.390563 14858 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:42.391098 14858 leader_election.cc:302] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d60bb7a14c448fa847919b31503477; no voters:
I20260619 06:30:42.391791 14858 leader_election.cc:288] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:30:42.391932 14862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:30:42.393302 14862 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 LEADER]: Becoming Leader. State: Replica: a2d60bb7a14c448fa847919b31503477, State: Running, Role: LEADER
I20260619 06:30:42.393884 14862 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:42.394248 14858 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:42.396508 14863 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a2d60bb7a14c448fa847919b31503477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } } }
I20260619 06:30:42.396915 14863 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:42.397223 14864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d60bb7a14c448fa847919b31503477. Latest consensus state: current_term: 2 leader_uuid: "a2d60bb7a14c448fa847919b31503477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } } }
I20260619 06:30:42.404386 14864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:42.409933 14869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:30:42.418543 14869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:30:42.419987 14869 catalog_manager.cc:1259] Loaded cluster ID: 8f5552e4b7c64979a8a08a9077a0fb49
I20260619 06:30:42.420122 14869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:30:42.423484 14869 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 06:30:42.441143 14869 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 06:30:42.450024 14869 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:30:42.451650 14869 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Loaded TSK: 0
I20260619 06:30:42.453075 14869 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:30:42.676901 14860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:42.677286 14860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:42.677457 14860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:42.684414 14860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:42.684546 14860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:42.684721 14860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:30:42.695091 14860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:38829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=43555
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:42.696331 14860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:42.697719 14860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:42.712201 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:42.712177 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:42.713449 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:42.713555 14860 server_base.cc:1034] running on GCE node
I20260619 06:30:42.714174 14860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:42.715152 14860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:42.716383 14860 hybrid_clock.cc:648] HybridClock initialized: now 1781850642716333 us; error 57 us; skew 500 ppm
I20260619 06:30:42.716709 14860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:42 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:42.720117 14860 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:30:42.721808 14860 webserver.cc:460] Webserver started at http://127.0.97.65:43555/ using document root <none> and password file <none>
I20260619 06:30:42.722476 14860 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:42.722635 14860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:42.728084 14860 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:30:42.730954 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:42.732324 14860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:30:42.732497 14860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "cdd1de95e6b045808f27aff9698adbd7"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "3538b62c478519a1ee4c70fb5f94ca0c"
server_key_iv: "e4254b95fa0f2a5d41af0b031a86440e"
server_key_version: "encryptionkey@0"
I20260619 06:30:42.733498 14860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:42.751542 14860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:42.755757 14860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:42.756021 14860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:42.756976 14860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:42.758361 14860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:42.758492 14860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:42.758623 14860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:42.758720 14860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:42.798945 14860 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:38829
I20260619 06:30:42.798990 15007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:38829 every 8 connection(s)
I20260619 06:30:42.800366 14860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:30:42.809537 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 14860
I20260619 06:30:42.810346 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:39175
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:35347
--webserver_port=41895
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:30:42 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:42.826604 15011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:42.803402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:52885)
0619 06:30:42.804245 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:42.804290 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:42.806067 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:42.807357 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:42.807365 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:42.807874 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:42.808406 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:42.808423 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:42.810176 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:42.810185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:42.812503 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:42.812522 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:42.813680 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:42.813691 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:42.814062 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:42.814823 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:42.814873 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:42.817932 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:42.822526 (+ 4594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:42.822533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:42.822544 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:42.822946 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:42.823426 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:42.823429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:42.823431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:42.823593 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:42.824526 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:42.824532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:42.824867 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 06:30:42.825214 (+ 347us) client_negotiation.cc:240] Negotiation successful
0619 06:30:42.825597 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":392,"threads_started":1}
I20260619 06:30:42.827111 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:42.803339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52885 (local address 127.0.97.126:35347)
0619 06:30:42.804011 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:42.804026 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:42.804424 (+ 398us) server_negotiation.cc:407] Connection header received
0619 06:30:42.806577 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:42.806590 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:42.806903 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:42.807187 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:42.808604 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:42.810015 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:42.812672 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:42.813439 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:42.813572 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:42.818235 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:42.818248 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:42.818269 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:42.818317 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:42.822384 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:42.823090 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:42.823101 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:42.823112 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:42.823311 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:42.823699 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:42.823706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:42.823712 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:42.824354 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:42.824532 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:42.825435 (+ 903us) server_negotiation.cc:299] Negotiation successful
0619 06:30:42.825871 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":255,"threads_started":1}
I20260619 06:30:42.830106 15008 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35347
I20260619 06:30:42.830449 15008 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:42.831395 15008 heartbeater.cc:511] Master 127.0.97.126:35347 requested a full tablet report, sending...
I20260619 06:30:42.834048 14823 ts_manager.cc:194] Registered new tserver with Master: cdd1de95e6b045808f27aff9698adbd7 (127.0.97.65:38829)
I20260619 06:30:42.836406 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:52885
W20260619 06:30:43.132283 15012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:43.132570 15012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:43.132663 15012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:43.139741 15012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:43.139838 15012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:43.139945 15012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:30:43.149130 15012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:39175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=41895
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:43.150252 15012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:43.151428 15012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:43.164011 15018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:43.164011 15017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:43.164074 15020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:43.165333 15012 server_base.cc:1034] running on GCE node
I20260619 06:30:43.165892 15012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:43.166811 15012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:43.168056 15012 hybrid_clock.cc:648] HybridClock initialized: now 1781850643167991 us; error 49 us; skew 500 ppm
I20260619 06:30:43.168378 15012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:43 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:43.171516 15012 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:30:43.172967 15012 webserver.cc:460] Webserver started at http://127.0.97.66:41895/ using document root <none> and password file <none>
I20260619 06:30:43.173709 15012 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:43.173905 15012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:43.179121 15012 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:30:43.181933 15027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:43.183193 15012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:43.183426 15012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "3051d8db01164019bbadb94f59ceda43"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "33220417311254b547b8406590a0dab8"
server_key_iv: "56b5edcc41ff80ea2f54597cd9f8f7e5"
server_key_version: "encryptionkey@0"
I20260619 06:30:43.184386 15012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:43.208923 15012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:43.212977 15012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:43.213265 15012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:43.214190 15012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:43.215472 15012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:43.215603 15012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:43.215736 15012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:43.215832 15012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:43.247916 15012 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39175
I20260619 06:30:43.247967 15139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39175 every 8 connection(s)
I20260619 06:30:43.249368 15012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:30:43.255378 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15012
I20260619 06:30:43.256136 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:36779
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:35347
--webserver_port=33769
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:30:43 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:43.272612 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.251857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51145 (local address 127.0.97.126:35347)
0619 06:30:43.252046 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.252051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.253389 (+ 1338us) server_negotiation.cc:407] Connection header received
0619 06:30:43.255966 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.255970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.256036 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.256118 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:43.258199 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.258910 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.261490 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.261914 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.262018 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.265819 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:43.265828 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:43.265836 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:43.265881 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:43.269402 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.270093 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.270101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.270107 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.270267 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.270682 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.270690 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.270696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.271085 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:43.271252 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.271891 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.272154 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:30:43.273239 15142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.252330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:51145)
0619 06:30:43.253238 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:43.253276 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:43.255674 (+ 2398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:43.256384 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:43.256397 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:43.257212 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:43.258008 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.258044 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.259085 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.259094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.261329 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.261346 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.262148 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.262158 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.262476 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.263200 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:43.263233 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:43.265584 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:43.269536 (+ 3952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:43.269543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:43.269555 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:43.269961 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:43.270375 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:43.270378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:43.270380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:43.270558 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:43.271203 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:43.271208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:43.271494 (+ 286us) client_negotiation.cc:769] Sending connection context
0619 06:30:43.271729 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 06:30:43.272103 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":353,"threads_started":1}
I20260619 06:30:43.274721 15140 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35347
I20260619 06:30:43.275048 15140 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:43.275731 15140 heartbeater.cc:511] Master 127.0.97.126:35347 requested a full tablet report, sending...
I20260619 06:30:43.277140 14823 ts_manager.cc:194] Registered new tserver with Master: 3051d8db01164019bbadb94f59ceda43 (127.0.97.66:39175)
I20260619 06:30:43.277937 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:51145
W20260619 06:30:43.582113 15143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:43.582406 15143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:43.582568 15143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:43.589922 15143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:43.590066 15143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:43.590231 15143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:30:43.600075 15143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:36779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=33769
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:43.601375 15143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:43.602648 15143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:43.616202 15149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:43.616225 15151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:43.617370 15148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:43.617652 15143 server_base.cc:1034] running on GCE node
I20260619 06:30:43.618229 15143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:43.619081 15143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:43.620266 15143 hybrid_clock.cc:648] HybridClock initialized: now 1781850643620205 us; error 59 us; skew 500 ppm
I20260619 06:30:43.620568 15143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:43 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:43.623759 15143 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:30:43.625275 15143 webserver.cc:460] Webserver started at http://127.0.97.67:33769/ using document root <none> and password file <none>
I20260619 06:30:43.625919 15143 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:43.626062 15143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:43.631069 15143 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:30:43.633744 15158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:43.635082 15143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:43.635253 15143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "221f10cb7eef44f196876a0f06deb3da"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "e74e15d98a3a1f1032d6fbce31dd7e75"
server_key_iv: "71f13618742f95ae227f0eac0a4f194a"
server_key_version: "encryptionkey@0"
I20260619 06:30:43.636188 15143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:43.664498 15143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:43.668651 15143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:43.668939 15143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:43.669986 15143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:43.671350 15143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:43.671466 15143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:43.671587 15143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:43.671675 15143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:43.713260 15143 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:36779
I20260619 06:30:43.713323 15270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:36779 every 8 connection(s)
I20260619 06:30:43.714649 15143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:30:43.721789 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15143
Jun 19 06:30:43 dist-test-slave-4t1f krb5kdc[14239](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850643, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:30:43.742321 15010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.717367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:37593 (local address 127.0.97.126:35347)
0619 06:30:43.717544 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.717548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.719071 (+ 1523us) server_negotiation.cc:407] Connection header received
0619 06:30:43.721092 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.721099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.721188 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.721298 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:43.723092 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.724622 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.727765 (+ 3143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.728713 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.728828 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.734637 (+ 5809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:43.734647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:43.734655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:43.734708 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:43.738877 (+ 4169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.739921 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.739929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.739935 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.740099 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.740597 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.740602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.740607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.740904 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:43.741067 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.741758 (+ 691us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.741977 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:30:43.743438 15273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.718056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:37593)
0619 06:30:43.718900 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:43.718942 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:43.720810 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:43.721552 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:43.721561 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:43.722111 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:30:43.722816 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.722839 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.724884 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.724892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.727553 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.727573 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.729266 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.729276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.729668 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.730604 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:30:43.730644 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:30:43.734339 (+ 3695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:30:43.739082 (+ 4743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:43.739093 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:43.739110 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:43.739787 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:43.740238 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:30:43.740242 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:30:43.740245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:30:43.740474 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:30:43.741043 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:30:43.741050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:30:43.741387 (+ 337us) client_negotiation.cc:769] Sending connection context
0619 06:30:43.741651 (+ 264us) client_negotiation.cc:240] Negotiation successful
0619 06:30:43.742131 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":377,"threads_started":1}
I20260619 06:30:43.744911 15271 heartbeater.cc:347] Connected to a master server at 127.0.97.126:35347
I20260619 06:30:43.745249 15271 heartbeater.cc:464] Registering TS with master...
I20260619 06:30:43.745942 15271 heartbeater.cc:511] Master 127.0.97.126:35347 requested a full tablet report, sending...
I20260619 06:30:43.747398 14823 ts_manager.cc:194] Registered new tserver with Master: 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779)
I20260619 06:30:43.748617 14823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:37593
I20260619 06:30:43.759218 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.722801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49892 (local address 127.0.97.126:35347)
0619 06:30:43.738210 (+ 15409us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.738218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.738242 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:30:43.738306 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.738313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.738395 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.738516 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:43.740455 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.742049 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.744278 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.745144 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.745291 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.751869 (+ 6578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:43.751878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:43.751885 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:43.751927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:43.755815 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.756642 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.756650 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.756656 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.756822 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.757347 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.757355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.757360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.757800 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:43.758085 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.758383 (+ 298us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.758666 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15258,"thread_start_us":15845,"threads_started":1}
I20260619 06:30:43.761862 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:30:43.770049 15274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.762857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.0.97.126:35347)
0619 06:30:43.763063 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.763074 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.763446 (+ 372us) server_negotiation.cc:407] Connection header received
0619 06:30:43.763604 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.763609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.763801 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.763967 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:30:43.764305 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.765589 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.767031 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.767907 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.768075 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.768552 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:30:43.769398 (+ 846us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:30:43.769562 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.769619 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.769714 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:30:43.771189 14823 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:30:43.783496 14823 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:30:43.808574 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.795346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44200 (local address 127.0.97.65:38829)
0619 06:30:43.795890 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.795903 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.797816 (+ 1913us) server_negotiation.cc:407] Connection header received
0619 06:30:43.798984 (+ 1168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.798998 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.799224 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.799417 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:43.800332 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.803003 (+ 2671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.805833 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.807362 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.807498 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.807703 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.808034 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.808214 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":303,"threads_started":1}
I20260619 06:30:43.808612 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.795277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44200)
0619 06:30:43.797545 (+ 2268us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:43.797596 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:43.797780 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:43.799461 (+ 1681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:43.799473 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:43.799516 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:43.800093 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.800109 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.803204 (+ 3095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.803212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.805637 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.805651 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.807560 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.807567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.807763 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.807822 (+ 59us) client_negotiation.cc:769] Sending connection context
0619 06:30:43.807954 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 06:30:43.808134 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2003,"mutex_wait_us":1,"thread_start_us":1835,"threads_started":1}
I20260619 06:30:43.809736 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.795764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49074 (local address 127.0.97.66:39175)
0619 06:30:43.796926 (+ 1162us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.796932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.797702 (+ 770us) server_negotiation.cc:407] Connection header received
0619 06:30:43.798041 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.798053 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.798319 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.798513 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:43.800413 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.801971 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.807004 (+ 5033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.808290 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.808411 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.808592 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.809231 (+ 639us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.809414 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1018,"thread_start_us":895,"threads_started":1}
I20260619 06:30:43.810541 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.795307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:39175 (local address 127.0.0.1:49074)
0619 06:30:43.797544 (+ 2237us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:43.797596 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:43.797792 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:43.799762 (+ 1970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:43.799767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:43.799792 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:43.800204 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.800215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.802700 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.802705 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.806855 (+ 4150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.806866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.808802 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.808809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.808991 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.809033 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:30:43.810029 (+ 996us) client_negotiation.cc:240] Negotiation successful
0619 06:30:43.810197 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2079,"mutex_wait_us":508,"thread_start_us":1954,"threads_started":1}
I20260619 06:30:43.812553 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.798539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51724 (local address 127.0.97.67:36779)
0619 06:30:43.804175 (+ 5636us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.804184 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.804899 (+ 715us) server_negotiation.cc:407] Connection header received
0619 06:30:43.805053 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.805113 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.805338 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.805525 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:43.806462 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.807905 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.810517 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.811620 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.811724 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.811869 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.812153 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.812311 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5468,"thread_start_us":5306,"threads_started":1}
I20260619 06:30:43.812557 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.797954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36779 (local address 127.0.0.1:51724)
0619 06:30:43.804766 (+ 6812us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:43.804792 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:43.804914 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:43.805809 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:43.805813 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:43.805838 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:43.806243 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.806255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.808128 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.808134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.810345 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.810359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.811833 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.811839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.811992 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.812025 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 06:30:43.812122 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:30:43.812253 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6697,"thread_start_us":6528,"threads_started":1}
I20260619 06:30:43.812811 14943 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76b72522894cb8997843d0ed85caee (DEFAULT_TABLE table=test-table [id=2fe54f03934d41358127aea679d56853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:43.814407 14943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76b72522894cb8997843d0ed85caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:43.815821 15206 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76b72522894cb8997843d0ed85caee (DEFAULT_TABLE table=test-table [id=2fe54f03934d41358127aea679d56853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:43.816848 15206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76b72522894cb8997843d0ed85caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:43.816433 15075 tablet_service.cc:1467] Processing CreateTablet for tablet 8a76b72522894cb8997843d0ed85caee (DEFAULT_TABLE table=test-table [id=2fe54f03934d41358127aea679d56853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:30:43.817658 15075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a76b72522894cb8997843d0ed85caee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:30:43.828405 15287 tablet_bootstrap.cc:492] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Bootstrap starting.
I20260619 06:30:43.831344 15287 tablet_bootstrap.cc:654] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:43.832124 15288 tablet_bootstrap.cc:492] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Bootstrap starting.
I20260619 06:30:43.832595 15287 log.cc:824] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:43.833619 15289 tablet_bootstrap.cc:492] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Bootstrap starting.
I20260619 06:30:43.836176 15288 tablet_bootstrap.cc:654] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:43.836784 15289 tablet_bootstrap.cc:654] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Neither blocks nor log segments found. Creating new log.
I20260619 06:30:43.837721 15288 log.cc:824] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:43.838049 15289 log.cc:824] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:43.839896 15008 heartbeater.cc:503] Master 127.0.97.126:35347 was elected leader, sending a full tablet report...
I20260619 06:30:43.840373 15287 tablet_bootstrap.cc:492] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: No bootstrap required, opened a new log
I20260619 06:30:43.840688 15287 ts_tablet_manager.cc:1397] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.007s
I20260619 06:30:43.841043 15289 tablet_bootstrap.cc:492] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: No bootstrap required, opened a new log
I20260619 06:30:43.841325 15289 ts_tablet_manager.cc:1397] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260619 06:30:43.848398 15288 tablet_bootstrap.cc:492] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: No bootstrap required, opened a new log
I20260619 06:30:43.848680 15288 ts_tablet_manager.cc:1397] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20260619 06:30:43.850281 15289 raft_consensus.cc:348] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.850700 15289 raft_consensus.cc:374] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:43.850874 15289 raft_consensus.cc:729] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3051d8db01164019bbadb94f59ceda43, State: Initialized, Role: FOLLOWER
I20260619 06:30:43.851502 15289 consensus_queue.cc:260] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.851527 15287 raft_consensus.cc:348] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.851840 15287 raft_consensus.cc:374] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:43.851958 15287 raft_consensus.cc:729] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 221f10cb7eef44f196876a0f06deb3da, State: Initialized, Role: FOLLOWER
I20260619 06:30:43.852593 15140 heartbeater.cc:503] Master 127.0.97.126:35347 was elected leader, sending a full tablet report...
I20260619 06:30:43.852655 15287 consensus_queue.cc:260] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.853291 15289 ts_tablet_manager.cc:1428] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260619 06:30:43.855475 15293 raft_consensus.cc:482] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:30:43.855777 15293 raft_consensus.cc:504] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.857640 15293 leader_election.cc:288] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdd1de95e6b045808f27aff9698adbd7 (127.0.97.65:38829), 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779)
I20260619 06:30:43.860489 15287 ts_tablet_manager.cc:1428] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Time spent starting tablet: real 0.020s user 0.010s sys 0.008s
I20260619 06:30:43.860976 15271 heartbeater.cc:503] Master 127.0.97.126:35347 was elected leader, sending a full tablet report...
I20260619 06:30:43.862294 15288 raft_consensus.cc:348] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.862761 15288 raft_consensus.cc:374] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:30:43.862900 15288 raft_consensus.cc:729] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdd1de95e6b045808f27aff9698adbd7, State: Initialized, Role: FOLLOWER
I20260619 06:30:43.863559 15288 consensus_queue.cc:260] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.867252 15288 ts_tablet_manager.cc:1428] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260619 06:30:43.867458 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.858229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46221 (local address 127.0.97.65:38829)
0619 06:30:43.858943 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.858952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.858977 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:30:43.859054 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.859060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.859143 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.859241 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:43.860583 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.861758 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.864507 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.866207 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.866321 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.866416 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.867007 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.867148 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535}
I20260619 06:30:43.868094 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.857837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.97.66:46221)
0619 06:30:43.858370 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:43.858391 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:43.858527 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:43.859961 (+ 1434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:43.859967 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:43.860008 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:43.860406 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.860418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.861923 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.861930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.864270 (+ 2340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.864294 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.866612 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.866620 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.866798 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.866844 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 06:30:43.867599 (+ 755us) client_negotiation.cc:240] Negotiation successful
0619 06:30:43.867746 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":437,"threads_started":1}
I20260619 06:30:43.868108 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.859185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36369 (local address 127.0.97.67:36779)
0619 06:30:43.859369 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.859376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.861036 (+ 1660us) server_negotiation.cc:407] Connection header received
0619 06:30:43.861312 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.861317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.861388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.861466 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:30:43.862314 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.863321 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.865708 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.866930 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.867055 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.867162 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.867704 (+ 542us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.867843 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:30:43.868775 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.858664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:36779 (local address 127.0.97.66:36369)
0619 06:30:43.860939 (+ 2275us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:43.860966 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0619 06:30:43.861146 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:30:43.861644 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:30:43.861648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:30:43.861676 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:30:43.862104 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.862121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.863494 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.863499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.865550 (+ 2051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:30:43.865564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.867312 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.867319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:30:43.867505 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.867541 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 06:30:43.868310 (+ 769us) client_negotiation.cc:240] Negotiation successful
0619 06:30:43.868445 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2149,"thread_start_us":1993,"threads_started":1}
I20260619 06:30:43.869474 14963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76b72522894cb8997843d0ed85caee" candidate_uuid: "3051d8db01164019bbadb94f59ceda43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd1de95e6b045808f27aff9698adbd7" is_pre_election: true
I20260619 06:30:43.869616 15226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76b72522894cb8997843d0ed85caee" candidate_uuid: "3051d8db01164019bbadb94f59ceda43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f10cb7eef44f196876a0f06deb3da" is_pre_election: true
I20260619 06:30:43.870002 14963 raft_consensus.cc:2454] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3051d8db01164019bbadb94f59ceda43 in term 0.
I20260619 06:30:43.870080 15226 raft_consensus.cc:2454] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3051d8db01164019bbadb94f59ceda43 in term 0.
I20260619 06:30:43.870667 15028 leader_election.cc:302] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3051d8db01164019bbadb94f59ceda43, cdd1de95e6b045808f27aff9698adbd7; no voters:
I20260619 06:30:43.871016 15293 raft_consensus.cc:2781] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:30:43.871160 15293 raft_consensus.cc:482] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:30:43.871265 15293 raft_consensus.cc:3037] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:43.873420 15293 raft_consensus.cc:504] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.874020 15293 leader_election.cc:288] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [CANDIDATE]: Term 1 election: Requested vote from peers cdd1de95e6b045808f27aff9698adbd7 (127.0.97.65:38829), 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779)
I20260619 06:30:43.874330 14963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76b72522894cb8997843d0ed85caee" candidate_uuid: "3051d8db01164019bbadb94f59ceda43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdd1de95e6b045808f27aff9698adbd7"
I20260619 06:30:43.874528 14963 raft_consensus.cc:3037] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:43.874495 15226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a76b72522894cb8997843d0ed85caee" candidate_uuid: "3051d8db01164019bbadb94f59ceda43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "221f10cb7eef44f196876a0f06deb3da"
I20260619 06:30:43.874693 15226 raft_consensus.cc:3037] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:30:43.877156 14963 raft_consensus.cc:2454] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3051d8db01164019bbadb94f59ceda43 in term 1.
I20260619 06:30:43.877228 15226 raft_consensus.cc:2454] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3051d8db01164019bbadb94f59ceda43 in term 1.
I20260619 06:30:43.877535 15028 leader_election.cc:302] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3051d8db01164019bbadb94f59ceda43, cdd1de95e6b045808f27aff9698adbd7; no voters:
I20260619 06:30:43.877805 15293 raft_consensus.cc:2781] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:30:43.878243 15293 raft_consensus.cc:686] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 1 LEADER]: Becoming Leader. State: Replica: 3051d8db01164019bbadb94f59ceda43, State: Running, Role: LEADER
I20260619 06:30:43.878697 15293 consensus_queue.cc:237] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } }
I20260619 06:30:43.883165 14823 catalog_manager.cc:5495] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3051d8db01164019bbadb94f59ceda43 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "3051d8db01164019bbadb94f59ceda43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3051d8db01164019bbadb94f59ceda43" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 39175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 } health_report { overall_health: UNKNOWN } } }
I20260619 06:30:43.906877 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.901939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.0.97.66:39175)
0619 06:30:43.902123 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.902127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.902206 (+ 79us) server_negotiation.cc:407] Connection header received
0619 06:30:43.902376 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.902379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.902474 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.902596 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:30:43.902905 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.903599 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.904908 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.905368 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.905468 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.905905 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:30:43.906371 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:30:43.906488 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.906547 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.906608 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:30:43.915160 15226 raft_consensus.cc:1261] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Refusing update from remote peer 3051d8db01164019bbadb94f59ceda43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:43.915160 14963 raft_consensus.cc:1261] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 1 FOLLOWER]: Refusing update from remote peer 3051d8db01164019bbadb94f59ceda43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:30:43.916133 15298 consensus_queue.cc:1035] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "221f10cb7eef44f196876a0f06deb3da" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 36779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:43.916499 15293 consensus_queue.cc:1035] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdd1de95e6b045808f27aff9698adbd7" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 38829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:30:43.930510 15303 mvcc.cc:205] Tried to move back new op lower bound from 7298460237465743360 to 7298460237331374080. Current Snapshot: MvccSnapshot[applied={T|T < 7298460237465743360}]
I20260619 06:30:43.930881 15302 mvcc.cc:205] Tried to move back new op lower bound from 7298460237465743360 to 7298460237331374080. Current Snapshot: MvccSnapshot[applied={T|T < 7298460237465743360}]
I20260619 06:30:43.946671 14822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:30:43.946997 14822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:30:43.950911 14822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Sending DeleteTablet for 3 replicas of tablet 8a76b72522894cb8997843d0ed85caee
I20260619 06:30:43.952126 14943 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a76b72522894cb8997843d0ed85caee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:43 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:44200
I20260619 06:30:43.952111 15206 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a76b72522894cb8997843d0ed85caee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:43 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:51724
I20260619 06:30:43.952361 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 14860
I20260619 06:30:43.953295 15075 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a76b72522894cb8997843d0ed85caee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:30:43 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49074
I20260619 06:30:43.953480 15312 tablet_replica.cc:330] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: stopping tablet replica
I20260619 06:30:43.953950 15312 raft_consensus.cc:2229] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:30:43.954049 15311 tablet_replica.cc:330] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: stopping tablet replica
I20260619 06:30:43.954517 15312 raft_consensus.cc:2258] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:43.955353 15313 tablet_replica.cc:330] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: stopping tablet replica
I20260619 06:30:43.955823 15313 raft_consensus.cc:2229] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:30:43.954427 15311 raft_consensus.cc:2229] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:30:43.956483 15313 raft_consensus.cc:2258] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:43.956802 15311 raft_consensus.cc:2258] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:30:43.957576 15312 ts_tablet_manager.cc:1905] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:43.959264 15313 ts_tablet_manager.cc:1905] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:43.964171 15311 ts_tablet_manager.cc:1905] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:43.964963 15312 ts_tablet_manager.cc:1918] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:43.965189 15312 log.cc:1196] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/8a76b72522894cb8997843d0ed85caee
I20260619 06:30:43.965750 15312 ts_tablet_manager.cc:1939] T 8a76b72522894cb8997843d0ed85caee P 221f10cb7eef44f196876a0f06deb3da: Deleting consensus metadata
I20260619 06:30:43.967608 14807 catalog_manager.cc:4841] TS 221f10cb7eef44f196876a0f06deb3da (127.0.97.67:36779): tablet 8a76b72522894cb8997843d0ed85caee (table test-table [id=2fe54f03934d41358127aea679d56853]) successfully deleted
I20260619 06:30:43.971028 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:43.954060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44210 (local address 127.0.97.65:38829)
0619 06:30:43.954229 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:43.954234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:43.954252 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:30:43.954307 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:43.954312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:43.954409 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:43.954534 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:43.956271 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.957538 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.959563 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:43.960195 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:43.960349 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:43.963696 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:43.963711 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:43.963726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:43.963766 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:43.967762 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.968650 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.968658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.968662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.968808 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:43.969253 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:43.969258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:43.969262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:43.969770 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:43.970065 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:43.970400 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 06:30:43.970634 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:30:43.971406 15313 ts_tablet_manager.cc:1918] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:43.971614 15313 log.cc:1196] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/8a76b72522894cb8997843d0ed85caee
I20260619 06:30:43.971752 15003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:43.972271 15313 ts_tablet_manager.cc:1939] T 8a76b72522894cb8997843d0ed85caee P 3051d8db01164019bbadb94f59ceda43: Deleting consensus metadata
I20260619 06:30:43.974553 14809 catalog_manager.cc:4841] TS 3051d8db01164019bbadb94f59ceda43 (127.0.97.66:39175): tablet 8a76b72522894cb8997843d0ed85caee (table test-table [id=2fe54f03934d41358127aea679d56853]) successfully deleted
I20260619 06:30:44.061883 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 14860
I20260619 06:30:44.082086 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 15012
W20260619 06:30:44.082238 14807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:38829 (error 108)
I20260619 06:30:44.083885 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.083491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44226)
0619 06:30:44.083654 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.083752 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:30:44.084193 14807 catalog_manager.cc:4584] TS cdd1de95e6b045808f27aff9698adbd7 (127.0.97.65:38829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a76b72522894cb8997843d0ed85caee: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
I20260619 06:30:44.099133 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.083555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.0.97.66:39175)
0619 06:30:44.083724 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:44.083731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:44.083749 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:30:44.083820 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:44.083826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:44.083913 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:44.084011 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:44.085942 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:44.086813 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:44.088825 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:44.089308 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:44.089405 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:44.092853 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:44.092868 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:44.092882 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:44.092921 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:44.096278 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:44.097050 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:44.097057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:44.097059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:44.097314 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:44.097741 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:44.097745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:44.097747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:44.098056 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:44.098295 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:44.098651 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:30:44.098818 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:30:44.099951 15135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:44.137787 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.137346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44230)
0619 06:30:44.137532 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.137619 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:30:44.138729 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.138412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44246)
0619 06:30:44.138554 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.138624 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:30:44.199530 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.199034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44258)
0619 06:30:44.199284 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.199367 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 06:30:44.200564 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.200252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44264)
0619 06:30:44.200393 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.200467 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260619 06:30:44.211215 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 15012
I20260619 06:30:44.230537 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 15143
I20260619 06:30:44.244189 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.231264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.0.97.67:36779)
0619 06:30:44.231450 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:30:44.231456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:30:44.231528 (+ 72us) server_negotiation.cc:407] Connection header received
0619 06:30:44.231780 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:30:44.231786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:30:44.231861 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:30:44.231973 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:30:44.233357 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:44.234099 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:44.235455 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:30:44.235862 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:30:44.236002 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:30:44.238678 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:30:44.238692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:30:44.238708 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:30:44.238745 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:30:44.241730 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:44.242357 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:44.242365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:44.242367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:44.242496 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:30:44.242843 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:30:44.242846 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:30:44.242848 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:30:44.243208 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:30:44.243406 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 06:30:44.243641 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:30:44.243792 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:30:44.244829 15266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:44.281163 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.280616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44272)
0619 06:30:44.280824 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.280919 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260619 06:30:44.282243 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.281845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44288)
0619 06:30:44.282064 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.282144 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 06:30:44.334681 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 15143
I20260619 06:30:44.356269 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 14790
I20260619 06:30:44.357530 14853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:30:44.469627 15285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:44.469132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:38829 (local address 127.0.0.1:44290)
0619 06:30:44.469314 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:44.469433 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:38829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:30:44.469676 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 14790
I20260619 06:30:44.491472 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35347
--webserver_interface=127.0.97.126
--webserver_port=35269
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:44.804093 15319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:44.804456 15319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:44.804610 15319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:44.811612 15319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:30:44.811756 15319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:44.811869 15319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:44.811971 15319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:30:44.812048 15319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:30:44.821681 15319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=35269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:44.822978 15319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:44.824265 15319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:44.835501 15324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:44.835466 15325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:44.835423 15327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:44.836380 15319 server_base.cc:1034] running on GCE node
I20260619 06:30:44.837234 15319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:44.838493 15319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:44.839710 15319 hybrid_clock.cc:648] HybridClock initialized: now 1781850644839668 us; error 28 us; skew 500 ppm
I20260619 06:30:44.840046 15319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:44 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850644, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:44.843070 15319 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:30:44.844502 15319 webserver.cc:460] Webserver started at http://127.0.97.126:35269/ using document root <none> and password file <none>
I20260619 06:30:44.845211 15319 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:44.845350 15319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:44.850351 15319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:30:44.852881 15334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:44.854135 15319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:30:44.854315 15319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "a2d60bb7a14c448fa847919b31503477"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "1f2a100f64d76334e546a5b1182bb0fa"
server_key_iv: "93252a986df8400cfe5ef97ba72bafd0"
server_key_version: "encryptionkey@0"
I20260619 06:30:44.855248 15319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:44.874338 15319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:44.878432 15319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:44.878707 15319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:44.896692 15319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35347
I20260619 06:30:44.896724 15385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35347 every 8 connection(s)
I20260619 06:30:44.898120 15319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:30:44.902163 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15319
I20260619 06:30:44.903086 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:38829
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:35347
--webserver_port=43555
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:30:44.905944 15386 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:30:44.911113 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Bootstrap starting.
I20260619 06:30:44.931419 15386 log.cc:824] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Log is configured to *not* fsync() on all Append() calls
I20260619 06:30:44.947129 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:30:44.947664 15386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477: Bootstrap complete.
I20260619 06:30:44.956594 15386 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:44.957521 15386 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d60bb7a14c448fa847919b31503477, State: Initialized, Role: FOLLOWER
I20260619 06:30:44.958194 15386 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:44.958453 15386 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:30:44.958603 15386 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:30:44.958778 15386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:30:44.962698 15386 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:44.963164 15386 leader_election.cc:302] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d60bb7a14c448fa847919b31503477; no voters:
I20260619 06:30:44.963832 15386 leader_election.cc:288] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:30:44.963966 15390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:30:44.966194 15386 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:30:44.971107 15390 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [term 3 LEADER]: Becoming Leader. State: Replica: a2d60bb7a14c448fa847919b31503477, State: Running, Role: LEADER
I20260619 06:30:44.971908 15390 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } }
I20260619 06:30:44.974850 15391 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a2d60bb7a14c448fa847919b31503477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } } }
I20260619 06:30:44.975243 15391 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:44.975878 15392 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d60bb7a14c448fa847919b31503477. Latest consensus state: current_term: 3 leader_uuid: "a2d60bb7a14c448fa847919b31503477" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d60bb7a14c448fa847919b31503477" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35347 } } }
I20260619 06:30:44.976182 15392 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d60bb7a14c448fa847919b31503477 [sys.catalog]: This master's current role is: LEADER
I20260619 06:30:44.978120 15402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:30:44.988703 15402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:30:44.990367 15402 catalog_manager.cc:1259] Loaded cluster ID: 8f5552e4b7c64979a8a08a9077a0fb49
I20260619 06:30:44.990513 15402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 06:30:44.992864 15402 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0474e789ad google::LogMessage::Fail() at ??:0
@ 0x7f0474e7b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0474e784de google::LogMessage::Flush() at ??:0
@ 0x7f0474e79a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f048a8ab512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f048a8a8976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260619 06:30:45.014577 15409 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f048a930610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f048a930439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0477ac4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0477ac791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0477ac7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0477aa2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f047ad7c6db start_thread at ??:0
@ 0x7f0473d1261f clone at ??:0
W20260619 06:30:45.265507 15388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:45.265897 15388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:45.266079 15388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:45.273643 15388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:45.273801 15388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:45.274009 15388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:30:45.284060 15388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:38829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=43555
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:45.285424 15388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:45.286845 15388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:45.300518 15431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:45.300660 15429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:45.300660 15428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:45.301522 15388 server_base.cc:1034] running on GCE node
I20260619 06:30:45.302219 15388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:45.303138 15388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:45.304368 15388 hybrid_clock.cc:648] HybridClock initialized: now 1781850645304320 us; error 30 us; skew 500 ppm
I20260619 06:30:45.304687 15388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:45 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:45.308086 15388 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:30:45.309603 15388 webserver.cc:460] Webserver started at http://127.0.97.65:43555/ using document root <none> and password file <none>
I20260619 06:30:45.310308 15388 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:45.310456 15388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:45.315929 15388 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:30:45.319015 15438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:45.320410 15388 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:30:45.320580 15388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "cdd1de95e6b045808f27aff9698adbd7"
format_stamp: "Formatted at 2026-06-19 06:30:39 on dist-test-slave-4t1f"
server_key: "3538b62c478519a1ee4c70fb5f94ca0c"
server_key_iv: "e4254b95fa0f2a5d41af0b031a86440e"
server_key_version: "encryptionkey@0"
I20260619 06:30:45.321687 15388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:45.357606 15388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:45.361588 15388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:45.361856 15388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:45.362809 15388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:45.364631 15445 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 06:30:45.370872 15445 ts_tablet_manager.cc:1852] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 06:30:45.371882 15445 ts_tablet_manager.cc:1905] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:30:45.378908 15445 ts_tablet_manager.cc:1918] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:30:45.379132 15445 log.cc:1196] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/8a76b72522894cb8997843d0ed85caee
I20260619 06:30:45.379580 15445 ts_tablet_manager.cc:1939] T 8a76b72522894cb8997843d0ed85caee P cdd1de95e6b045808f27aff9698adbd7: Deleting consensus metadata
I20260619 06:30:45.379981 15388 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 06:30:45.380209 15388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260619 06:30:45.380360 15388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:45.380451 15388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:45.413312 15388 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:38829
I20260619 06:30:45.413396 15551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:38829 every 8 connection(s)
I20260619 06:30:45.414590 15388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:30:45.418570 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15388
I20260619 06:30:45.419198 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:39175
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:35347
--webserver_port=41895
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:30:45.419456 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:45.417539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:51051)
0619 06:30:45.418264 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:45.418479 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":271,"threads_started":1}
W20260619 06:30:45.420075 15552 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35347 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:35347: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
I20260619 06:30:45.420931 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:45.420612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:54067)
0619 06:30:45.420753 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:45.420828 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:30:45.421744 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:45.421428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:59159)
0619 06:30:45.421560 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:45.421639 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:30:45.421968 15552 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:30:45.734815 15555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:45.735090 15555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:45.735265 15555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:45.742230 15555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:45.742353 15555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:45.742540 15555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:30:45.751775 15555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:39175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=41895
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:45.752943 15555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:45.754185 15555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:45.766927 15561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:45.767005 15560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:45.767027 15563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:45.767959 15555 server_base.cc:1034] running on GCE node
I20260619 06:30:45.768522 15555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:45.769546 15555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:45.770763 15555 hybrid_clock.cc:648] HybridClock initialized: now 1781850645770716 us; error 62 us; skew 500 ppm
I20260619 06:30:45.771075 15555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:45 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:45.774112 15555 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:30:45.775501 15555 webserver.cc:460] Webserver started at http://127.0.97.66:41895/ using document root <none> and password file <none>
I20260619 06:30:45.776161 15555 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:45.776302 15555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:45.781612 15555 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:30:45.784104 15570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:45.785414 15555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:30:45.785579 15555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "3051d8db01164019bbadb94f59ceda43"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "33220417311254b547b8406590a0dab8"
server_key_iv: "56b5edcc41ff80ea2f54597cd9f8f7e5"
server_key_version: "encryptionkey@0"
I20260619 06:30:45.786537 15555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:45.801417 15555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:45.804875 15555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:45.805205 15555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:45.806090 15555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:45.807296 15555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:45.807420 15555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:45.807550 15555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:45.807641 15555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:45.838877 15555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:39175
I20260619 06:30:45.838970 15682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:39175 every 8 connection(s)
I20260619 06:30:45.840390 15555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:30:45.844834 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:45.843185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:34163)
0619 06:30:45.843858 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:45.844050 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":273,"threads_started":1}
W20260619 06:30:45.845482 15683 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35347 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:35347: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
I20260619 06:30:45.846191 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:45.845900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:56951)
0619 06:30:45.846039 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:45.846099 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:30:45.846935 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:45.846651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:57619)
0619 06:30:45.846767 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:45.846834 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:30:45.847168 15683 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:30:45.850338 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15555
I20260619 06:30:45.850934 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:36779
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:35347
--webserver_port=33769
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:30:46.173348 15686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:30:46.173633 15686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:30:46.173840 15686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:30:46.181288 15686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:30:46.181416 15686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:30:46.181623 15686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:30:46.191286 15686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:36779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=33769
--tserver_master_addrs=127.0.97.126:35347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:30:46.192600 15686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:30:46.193946 15686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:30:46.206826 15694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:46.206861 15691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:30:46.206897 15692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:30:46.207696 15686 server_base.cc:1034] running on GCE node
I20260619 06:30:46.208423 15686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:30:46.209442 15686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:30:46.210664 15686 hybrid_clock.cc:648] HybridClock initialized: now 1781850646210632 us; error 49 us; skew 500 ppm
I20260619 06:30:46.210955 15686 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:30:46 dist-test-slave-4t1f krb5kdc[14239](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:30:46.213927 15686 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:30:46.215235 15686 webserver.cc:460] Webserver started at http://127.0.97.67:33769/ using document root <none> and password file <none>
I20260619 06:30:46.215869 15686 fs_manager.cc:359] Metadata directory not provided
I20260619 06:30:46.216009 15686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:30:46.221240 15686 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:30:46.223740 15701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:46.225116 15686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:30:46.225282 15686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "221f10cb7eef44f196876a0f06deb3da"
format_stamp: "Formatted at 2026-06-19 06:30:40 on dist-test-slave-4t1f"
server_key: "e74e15d98a3a1f1032d6fbce31dd7e75"
server_key_iv: "71f13618742f95ae227f0eac0a4f194a"
server_key_version: "encryptionkey@0"
I20260619 06:30:46.226265 15686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:30:46.264740 15686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:30:46.269160 15686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:30:46.269443 15686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:30:46.270403 15686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:30:46.271664 15686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:30:46.271785 15686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:46.271920 15686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:30:46.272006 15686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:30:46.304301 15686 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:36779
I20260619 06:30:46.304383 15813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:36779 every 8 connection(s)
I20260619 06:30:46.305719 15686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:30:46.310319 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:46.308531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:34491)
0619 06:30:46.309295 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:46.309508 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":267,"threads_started":1}
W20260619 06:30:46.310956 15814 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:35347 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:35347: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
I20260619 06:30:46.311777 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:46.311415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:52775)
0619 06:30:46.311560 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:46.311644 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:30:46.312017 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15686
I20260619 06:30:46.312700 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:46.312347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:51295)
0619 06:30:46.312501 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:46.312594 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 06:30:46.312924 15814 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:30:46.423235 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:46.422414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:58831)
0619 06:30:46.422868 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:46.423009 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":201,"threads_started":1}
I20260619 06:30:46.848456 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:46.847610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:35625)
0619 06:30:46.848148 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:46.848248 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":310,"threads_started":1}
I20260619 06:30:47.314235 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:47.313451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:58145)
0619 06:30:47.313904 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:47.314007 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":217,"threads_started":1}
I20260619 06:30:47.424618 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:47.424017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:37715)
0619 06:30:47.424400 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:47.424474 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":203,"threads_started":1}
I20260619 06:30:47.849983 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:47.849309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:42413)
0619 06:30:47.849747 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:47.849826 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":205,"threads_started":1}
I20260619 06:30:48.315761 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:48.315089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:54607)
0619 06:30:48.315512 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:48.315598 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":219,"threads_started":1}
I20260619 06:30:48.426311 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:48.425468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:52513)
0619 06:30:48.426046 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:48.426133 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":329,"threads_started":1}
I20260619 06:30:48.851454 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:48.850785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:38915)
0619 06:30:48.851184 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:48.851260 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
I20260619 06:30:49.317293 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:49.316533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:37839)
0619 06:30:49.317003 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:49.317130 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":288,"threads_started":1}
I20260619 06:30:49.427874 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:49.427222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:37029)
0619 06:30:49.427636 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:49.427709 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":217,"threads_started":1}
I20260619 06:30:49.852994 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:49.852302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:36533)
0619 06:30:49.852729 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:49.852811 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":205,"threads_started":1}
I20260619 06:30:50.318917 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:50.318248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:48217)
0619 06:30:50.318684 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:50.318765 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":212,"threads_started":1}
I20260619 06:30:50.429528 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:50.428820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:39575)
0619 06:30:50.429265 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:50.429348 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":252,"threads_started":1}
I20260619 06:30:50.854908 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:50.854156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:54823)
0619 06:30:50.854657 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:50.854745 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":254,"threads_started":1}
I20260619 06:30:51.320719 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:51.319935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:57857)
0619 06:30:51.320403 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:51.320494 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":219,"threads_started":1}
I20260619 06:30:51.431300 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:51.430470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:40265)
0619 06:30:51.431057 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:51.431146 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":287,"threads_started":1}
I20260619 06:30:51.856642 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:51.855886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:40691)
0619 06:30:51.856386 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:51.856476 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":276,"threads_started":1}
I20260619 06:30:52.322263 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:52.321548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:45815)
0619 06:30:52.322016 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:52.322104 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":234,"threads_started":1}
I20260619 06:30:52.432919 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:52.432172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:48379)
0619 06:30:52.432645 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:52.432723 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":221,"threads_started":1}
I20260619 06:30:52.858295 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:52.857571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:59683)
0619 06:30:52.858019 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:52.858093 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":197,"threads_started":1}
I20260619 06:30:53.323830 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:53.323089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:58809)
0619 06:30:53.323553 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:53.323629 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":197,"threads_started":1}
I20260619 06:30:53.434425 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:53.433772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:53909)
0619 06:30:53.434185 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:53.434262 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":218,"threads_started":1}
I20260619 06:30:53.859887 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:53.859197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:38775)
0619 06:30:53.859607 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:53.859682 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":217,"threads_started":1}
I20260619 06:30:54.325332 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:54.324649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:52591)
0619 06:30:54.325059 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:54.325185 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":216,"threads_started":1}
I20260619 06:30:54.435973 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:54.435244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:53765)
0619 06:30:54.435698 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:54.435804 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":231,"threads_started":1}
I20260619 06:30:54.861543 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:54.860817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:49587)
0619 06:30:54.861305 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:54.861381 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":300,"threads_started":1}
I20260619 06:30:55.326925 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:55.326158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:55593)
0619 06:30:55.326661 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:55.326756 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":216,"threads_started":1}
I20260619 06:30:55.437635 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:55.436921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:43929)
0619 06:30:55.437397 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:55.437476 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":279,"threads_started":1}
I20260619 06:30:55.863078 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:55.862359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:39671)
0619 06:30:55.862852 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:55.862927 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":289,"threads_started":1}
I20260619 06:30:56.328352 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:56.327726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:35045)
0619 06:30:56.328130 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:56.328204 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":219,"threads_started":1}
I20260619 06:30:56.439188 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:56.438477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:44279)
0619 06:30:56.438910 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:56.439005 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":243,"threads_started":1}
I20260619 06:30:56.864552 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:56.863886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:58261)
0619 06:30:56.864296 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:56.864381 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":211,"threads_started":1}
I20260619 06:30:57.329924 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:57.329204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:60719)
0619 06:30:57.329664 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:57.329749 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
I20260619 06:30:57.440654 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:57.440002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:33603)
0619 06:30:57.440428 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:57.440504 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":226,"threads_started":1}
I20260619 06:30:57.866458 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:57.865591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:44313)
0619 06:30:57.866145 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:57.866280 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":281,"threads_started":1}
I20260619 06:30:58.331395 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:58.330713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:36647)
0619 06:30:58.331145 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:58.331221 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260619 06:30:58.442333 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:58.441617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:38833)
0619 06:30:58.442019 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:58.442112 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":203,"threads_started":1}
I20260619 06:30:58.868103 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:58.867345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:56549)
0619 06:30:58.867826 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:58.867914 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":241,"threads_started":1}
I20260619 06:30:59.332883 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:59.332181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:53693)
0619 06:30:59.332613 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:59.332703 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":217,"threads_started":1}
I20260619 06:30:59.443887 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:59.443189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:34147)
0619 06:30:59.443616 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:59.443696 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20260619 06:30:59.869666 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:30:59.868899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:34853)
0619 06:30:59.869410 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0619 06:30:59.869489 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":299,"threads_started":1}
I20260619 06:31:00.334451 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:00.333796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:56785)
0619 06:31:00.334202 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:00.334277 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":197,"threads_started":1}
I20260619 06:31:00.445660 15860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:00.444937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.65:40369)
0619 06:31:00.445399 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:00.445483 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":261,"threads_started":1}
I20260619 06:31:00.871412 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:00.870690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.66:33971)
0619 06:31:00.871136 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:00.871232 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260619 06:31:01.314698 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 15388
I20260619 06:31:01.335081 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:01.315951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.97.65:38829)
0619 06:31:01.317027 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:01.317051 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:01.317127 (+ 76us) server_negotiation.cc:407] Connection header received
0619 06:31:01.320236 (+ 3109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:01.320254 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:01.320692 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:01.321102 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:01.322638 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:01.323978 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:01.325368 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:01.326169 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:01.326376 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:01.329003 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:01.329011 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:01.329026 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:01.329119 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:01.332018 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:01.332662 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:01.332668 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:01.332675 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:01.332834 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:01.333269 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:01.333272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:01.333274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:01.333785 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:01.334009 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:01.334381 (+ 372us) server_negotiation.cc:299] Negotiation successful
0619 06:31:01.334798 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":1284,"threads_started":1}
I20260619 06:31:01.336277 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:01.335525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:35347 (local address 127.0.97.67:49705)
0619 06:31:01.336019 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:01.336112 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":269,"threads_started":1}
I20260619 06:31:01.336599 15547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:01.422231 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 15388
I20260619 06:31:01.439559 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 15555
I20260619 06:31:01.456800 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:01.440642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.0.97.66:39175)
0619 06:31:01.441293 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:01.441308 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:01.441344 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:31:01.442898 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:01.442907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:01.443328 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:01.443620 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:01.444855 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:01.446186 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:01.447347 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:01.448001 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:01.448132 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:01.450762 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:01.450770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:01.450785 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:01.450821 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:01.453932 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:01.454552 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:01.454558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:01.454564 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:01.454716 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:01.455028 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:01.455056 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:01.455058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:01.455525 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:01.455735 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:01.456114 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 06:31:01.456514 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":300,"threads_started":1}
I20260619 06:31:01.458175 15678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:01.540642 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 15555
I20260619 06:31:01.557312 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 15686
I20260619 06:31:01.574656 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:01.558281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.0.97.67:36779)
0619 06:31:01.558907 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:01.558920 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:01.558961 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:31:01.560592 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:01.560601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:01.561022 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:01.561401 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:01.562665 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:01.563869 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:01.565039 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:01.565849 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:01.565989 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:01.568506 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:01.568514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:01.568528 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:01.568582 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:01.571588 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:01.572252 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:01.572259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:01.572265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:01.572426 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:01.572745 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:01.572748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:01.572750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:01.573348 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:01.573577 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:01.573903 (+ 326us) server_negotiation.cc:299] Negotiation successful
0619 06:31:01.574324 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":331,"threads_started":1}
I20260619 06:31:01.575995 15809 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:01.658243 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 15686
2026-06-19T06:31:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24215 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:31:01 dist-test-slave-4t1f krb5kdc[15871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:31:01 dist-test-slave-4t1f krb5kdc[15871](info): set up 2 sockets
Jun 19 06:31:01 dist-test-slave-4t1f krb5kdc[15871](info): commencing operation
krb5kdc: starting...
W20260619 06:31:02.757634 389 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:31:02 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:31:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:31:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:31:02.992151 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40179
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:31:03.311918 15884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:03.312196 15884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:03.312361 15884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:03.319526 15884 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:31:03.319638 15884 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:31:03.319728 15884 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:31:03.319820 15884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:31:03.319909 15884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:03.320031 15884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:03.320107 15884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:31:03.320184 15884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:31:03.331450 15884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:03.332723 15884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:03.334120 15884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:03.345880 15892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:03.345837 15889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:03.345851 15890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:03.346557 15884 server_base.cc:1034] running on GCE node
I20260619 06:31:03.347307 15884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:03.348657 15884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:03.349900 15884 hybrid_clock.cc:648] HybridClock initialized: now 1781850663349844 us; error 50 us; skew 500 ppm
I20260619 06:31:03.350224 15884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:03 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:03.353370 15884 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:31:03.354676 15884 webserver.cc:460] Webserver started at http://127.0.97.126:39731/ using document root <none> and password file <none>
I20260619 06:31:03.355302 15884 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:03.355384 15884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:03.355639 15884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:31:03.358486 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "bb19f564665a46c09f121f1b7375cc29"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "4e8b2a3beb1165972e83ec79502a3d58"
server_key_iv: "854989160e0242d5f3e4e7c2d1f21744"
server_key_version: "encryptionkey@0"
I20260619 06:31:03.359196 15884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "bb19f564665a46c09f121f1b7375cc29"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "4e8b2a3beb1165972e83ec79502a3d58"
server_key_iv: "854989160e0242d5f3e4e7c2d1f21744"
server_key_version: "encryptionkey@0"
I20260619 06:31:03.364207 15884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:31:03.367524 15899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:03.368831 15884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:31:03.368999 15884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "bb19f564665a46c09f121f1b7375cc29"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "4e8b2a3beb1165972e83ec79502a3d58"
server_key_iv: "854989160e0242d5f3e4e7c2d1f21744"
server_key_version: "encryptionkey@0"
I20260619 06:31:03.369311 15884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:03.390196 15884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:03.394361 15884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:03.394644 15884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:03.412631 15950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:40179 every 8 connection(s)
I20260619 06:31:03.412659 15884 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:40179
I20260619 06:31:03.414050 15884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:31:03.417853 15951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:31:03.423105 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15884
I20260619 06:31:03.423337 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:31:03.423971 389 external_mini_cluster.cc:1428] Setting key 64a10011c13b4fbd04a9c6537a001772
I20260619 06:31:03.428471 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Bootstrap starting.
I20260619 06:31:03.432140 15951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Neither blocks nor log segments found. Creating new log.
I20260619 06:31:03.433418 15951 log.cc:824] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:03.436496 15951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: No bootstrap required, opened a new log
Jun 19 06:31:03 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:31:03.446954 15951 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:03.447381 15951 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:31:03.447489 15951 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb19f564665a46c09f121f1b7375cc29, State: Initialized, Role: FOLLOWER
I20260619 06:31:03.448149 15951 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:03.448426 15951 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:31:03.448551 15954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:03.425904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.97.126:40179)
0619 06:31:03.426741 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:03.426752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:03.426790 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:31:03.428532 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:03.428554 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:03.428972 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:03.429349 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:03.431117 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:03.432471 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:03.434411 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:03.435179 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:03.435308 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:03.440212 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:03.440224 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:03.440245 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:03.440303 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:03.444283 (+ 3980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:03.444988 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:03.444998 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:03.445006 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:03.445232 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:03.445608 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:03.445613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:03.445617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:03.446279 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:03.446561 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:03.446831 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 06:31:03.447242 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":416,"threads_started":1}
I20260619 06:31:03.448578 15951 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:31:03.448745 15951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:31:03.451071 15951 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:03.451535 15951 leader_election.cc:302] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb19f564665a46c09f121f1b7375cc29; no voters:
I20260619 06:31:03.452090 15951 leader_election.cc:288] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:31:03.452229 15956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:31:03.452894 15956 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 LEADER]: Becoming Leader. State: Replica: bb19f564665a46c09f121f1b7375cc29, State: Running, Role: LEADER
I20260619 06:31:03.453404 15956 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:03.454036 15951 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:31:03.456075 15958 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb19f564665a46c09f121f1b7375cc29. Latest consensus state: current_term: 1 leader_uuid: "bb19f564665a46c09f121f1b7375cc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } } }
I20260619 06:31:03.456388 15958 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:31:03.460225 15957 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb19f564665a46c09f121f1b7375cc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } } }
I20260619 06:31:03.460695 15957 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:31:03.461585 15963 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:31:03.466573 15963 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:31:03.475515 15963 catalog_manager.cc:1347] Generated new cluster ID: 0358a5768b594044b038b5c56d679988
I20260619 06:31:03.475759 15963 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:31:03.492645 15963 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:31:03.494117 15963 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:31:03.501879 15963 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Generated new TSK 0
I20260619 06:31:03.502743 15963 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:31:03.557921 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40179
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:31:03.873960 15977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:03.874286 15977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:03.874457 15977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:03.881755 15977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:03.881891 15977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:03.882050 15977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:31:03.891328 15977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:03.892516 15977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:03.893807 15977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:03.906229 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:03.906497 15985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:03.906520 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:03.907524 15977 server_base.cc:1034] running on GCE node
I20260619 06:31:03.908085 15977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:03.908939 15977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:03.910164 15977 hybrid_clock.cc:648] HybridClock initialized: now 1781850663910132 us; error 54 us; skew 500 ppm
I20260619 06:31:03.910460 15977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:03 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:03.913542 15977 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:31:03.914830 15977 webserver.cc:460] Webserver started at http://127.0.97.65:35983/ using document root <none> and password file <none>
I20260619 06:31:03.915403 15977 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:03.915495 15977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:03.915742 15977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:31:03.918478 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "a7e868380c534359b78cac570b48a3da"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "fb1d3e03abe9aaec0e5f3e3ab1af78ed"
server_key_iv: "eaaf2c3a7a28182a7f19c9e6f65cf4a6"
server_key_version: "encryptionkey@0"
I20260619 06:31:03.919145 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "a7e868380c534359b78cac570b48a3da"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "fb1d3e03abe9aaec0e5f3e3ab1af78ed"
server_key_iv: "eaaf2c3a7a28182a7f19c9e6f65cf4a6"
server_key_version: "encryptionkey@0"
I20260619 06:31:03.923839 15977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:31:03.926955 15992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:03.928328 15977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:31:03.928462 15977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "a7e868380c534359b78cac570b48a3da"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "fb1d3e03abe9aaec0e5f3e3ab1af78ed"
server_key_iv: "eaaf2c3a7a28182a7f19c9e6f65cf4a6"
server_key_version: "encryptionkey@0"
I20260619 06:31:03.928623 15977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:03.940286 15977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:03.944650 15977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:03.944957 15977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:03.946125 15977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:03.947535 15977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:03.947639 15977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:03.947800 15977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:03.947861 15977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:03.980326 15977 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41525
I20260619 06:31:03.980361 16104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41525 every 8 connection(s)
I20260619 06:31:03.981680 15977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:31:03.990658 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 15977
I20260619 06:31:03.990872 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:31:03.991496 389 external_mini_cluster.cc:1428] Setting key d137142981c380c6247514109b8552c7
Jun 19 06:31:03 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:31:04.001030 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:03.984206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:45811 (local address 127.0.97.126:40179)
0619 06:31:03.984616 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:03.984621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:03.985619 (+ 998us) server_negotiation.cc:407] Connection header received
0619 06:31:03.987273 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:03.987278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:03.987342 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:03.987435 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:03.988749 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:03.989529 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:03.991035 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:03.991404 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:03.991535 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:03.995158 (+ 3623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:03.995162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:03.995165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:03.995207 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:03.997842 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:03.998570 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:03.998575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:03.998579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:03.998696 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:03.999120 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:03.999127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:03.999132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:03.999399 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:03.999557 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:04.000212 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 06:31:04.000424 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":207,"threads_started":1}
I20260619 06:31:04.002131 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:03.984739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:45811)
0619 06:31:03.985465 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:03.985505 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:03.987025 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:03.987629 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:03.987636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:03.988099 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:31:03.988608 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:03.988624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:03.989662 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:03.989666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:03.990866 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:03.990881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:03.991612 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:03.991619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:03.991849 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:03.992485 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:31:03.992516 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:31:03.994944 (+ 2428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:31:03.998019 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:03.998027 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:03.998042 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:03.998447 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:03.998804 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:03.998810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:03.998814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:03.998998 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:03.999519 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:31:03.999527 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:31:03.999862 (+ 335us) client_negotiation.cc:769] Sending connection context
0619 06:31:04.000123 (+ 261us) client_negotiation.cc:240] Negotiation successful
0619 06:31:04.000620 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":323,"threads_started":1}
I20260619 06:31:04.003654 16105 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40179
I20260619 06:31:04.003983 16105 heartbeater.cc:464] Registering TS with master...
I20260619 06:31:04.004768 16105 heartbeater.cc:511] Master 127.0.97.126:40179 requested a full tablet report, sending...
I20260619 06:31:04.006846 15916 ts_manager.cc:194] Registered new tserver with Master: a7e868380c534359b78cac570b48a3da (127.0.97.65:41525)
I20260619 06:31:04.008275 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:45811
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:31:04.042212 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40179
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:31:04.376029 16111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:04.376340 16111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:04.376569 16111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:04.384430 16111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:04.384558 16111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:04.384765 16111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:31:04.394907 16111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:04.396298 16111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:04.397728 16111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:04.411321 16117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:04.411453 16116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:04.411520 16119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:04.413296 16111 server_base.cc:1034] running on GCE node
I20260619 06:31:04.413954 16111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:04.414888 16111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:04.416126 16111 hybrid_clock.cc:648] HybridClock initialized: now 1781850664416070 us; error 52 us; skew 500 ppm
I20260619 06:31:04.416481 16111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:04 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:04.419940 16111 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:31:04.421579 16111 webserver.cc:460] Webserver started at http://127.0.97.66:38447/ using document root <none> and password file <none>
I20260619 06:31:04.422297 16111 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:04.422434 16111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:04.422765 16111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:31:04.425757 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "406d763d570246fbaa2549e3144e8a7f"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "afb88a578d9494b26736b8d07e25f4f5"
server_key_iv: "4c647fd2e3c34de10effb31b380258ae"
server_key_version: "encryptionkey@0"
I20260619 06:31:04.426674 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "406d763d570246fbaa2549e3144e8a7f"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "afb88a578d9494b26736b8d07e25f4f5"
server_key_iv: "4c647fd2e3c34de10effb31b380258ae"
server_key_version: "encryptionkey@0"
I20260619 06:31:04.432282 16111 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:31:04.435947 16126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:04.437470 16111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:31:04.437645 16111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "406d763d570246fbaa2549e3144e8a7f"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "afb88a578d9494b26736b8d07e25f4f5"
server_key_iv: "4c647fd2e3c34de10effb31b380258ae"
server_key_version: "encryptionkey@0"
I20260619 06:31:04.437853 16111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:04.453238 16111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:04.457568 16111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:04.457860 16111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:04.458993 16111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:04.460420 16111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:04.460500 16111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:04.460655 16111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:04.460711 16111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:04.495141 16111 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37251
I20260619 06:31:04.495191 16238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37251 every 8 connection(s)
I20260619 06:31:04.496510 16111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:31:04.503389 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 16111
I20260619 06:31:04.503607 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:31:04.504076 389 external_mini_cluster.cc:1428] Setting key 8592a07da7bebe984d1c92fa540fdedf
Jun 19 06:31:04 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:31:04.523331 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:04.499258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60399 (local address 127.0.97.126:40179)
0619 06:31:04.499450 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:04.499454 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:04.500704 (+ 1250us) server_negotiation.cc:407] Connection header received
0619 06:31:04.503023 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:04.503031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:04.503131 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:04.503250 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:04.505188 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:04.506067 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:04.508677 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:04.509150 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:04.509251 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:04.516153 (+ 6902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:04.516163 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:04.516172 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:04.516221 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:04.519906 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:04.520674 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:04.520683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:04.520690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:04.520858 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:04.521347 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:04.521355 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:04.521361 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:04.521740 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:04.521893 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:04.522591 (+ 698us) server_negotiation.cc:299] Negotiation successful
0619 06:31:04.522858 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:31:04.523988 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:04.499787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:60399)
0619 06:31:04.500518 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:04.500561 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:04.502692 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:04.503516 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:04.503525 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:04.504200 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:31:04.504971 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:04.504993 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:04.506255 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:04.506260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:04.508514 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:04.508535 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:04.509390 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:04.509400 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:04.509746 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:04.513492 (+ 3746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:31:04.513531 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:31:04.515878 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:31:04.520069 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:04.520075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:04.520090 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:04.520528 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:04.520971 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:04.520974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:04.520976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:04.521218 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:04.521863 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:31:04.521869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:31:04.522202 (+ 333us) client_negotiation.cc:769] Sending connection context
0619 06:31:04.522427 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 06:31:04.522830 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":296,"threads_started":1}
I20260619 06:31:04.525676 16239 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40179
I20260619 06:31:04.526023 16239 heartbeater.cc:464] Registering TS with master...
I20260619 06:31:04.526994 16239 heartbeater.cc:511] Master 127.0.97.126:40179 requested a full tablet report, sending...
I20260619 06:31:04.528684 15916 ts_manager.cc:194] Registered new tserver with Master: 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251)
I20260619 06:31:04.529736 15916 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:60399
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:31:04.559719 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40179
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:31:04.888154 16244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:04.888458 16244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:04.888559 16244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:04.895835 16244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:04.895951 16244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:04.896087 16244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:31:04.906293 16244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:04.907558 16244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:04.908854 16244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:04.922164 16250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:04.922101 16252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:04.922670 16249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:04.923199 16244 server_base.cc:1034] running on GCE node
I20260619 06:31:04.923712 16244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:04.924625 16244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:04.925873 16244 hybrid_clock.cc:648] HybridClock initialized: now 1781850664925833 us; error 49 us; skew 500 ppm
I20260619 06:31:04.926191 16244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:04 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:04.929437 16244 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:31:04.931025 16244 webserver.cc:460] Webserver started at http://127.0.97.67:42217/ using document root <none> and password file <none>
I20260619 06:31:04.931686 16244 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:04.931838 16244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:04.932232 16244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:31:04.935232 16244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "18af0be18e764bab9958b7336d6689ba"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "895adf0e60cad5e92f2e44d8d8716bcf"
server_key_iv: "845b3d9f3a548ee86cfad4732f01770e"
server_key_version: "encryptionkey@0"
I20260619 06:31:04.936010 16244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "18af0be18e764bab9958b7336d6689ba"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "895adf0e60cad5e92f2e44d8d8716bcf"
server_key_iv: "845b3d9f3a548ee86cfad4732f01770e"
server_key_version: "encryptionkey@0"
I20260619 06:31:04.941205 16244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:31:04.944633 16259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:04.945963 16244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:31:04.946128 16244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "18af0be18e764bab9958b7336d6689ba"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "895adf0e60cad5e92f2e44d8d8716bcf"
server_key_iv: "845b3d9f3a548ee86cfad4732f01770e"
server_key_version: "encryptionkey@0"
I20260619 06:31:04.946328 16244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:04.967025 16244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:04.970690 16244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:04.970964 16244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:04.971880 16244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:04.973198 16244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:04.973299 16244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:04.973430 16244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:04.973508 16244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:05.006497 16244 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39185
I20260619 06:31:05.006582 16371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39185 every 8 connection(s)
I20260619 06:31:05.007807 16244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:31:05.011318 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 16244
I20260619 06:31:05.011536 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:31:05.011745 16105 heartbeater.cc:503] Master 127.0.97.126:40179 was elected leader, sending a full tablet report...
I20260619 06:31:05.012156 389 external_mini_cluster.cc:1428] Setting key a370f5244ae0ffc305046ef2f25b41e5
Jun 19 06:31:05 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:31:05.028914 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.010417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45511 (local address 127.0.97.126:40179)
0619 06:31:05.010654 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.010658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.012694 (+ 2036us) server_negotiation.cc:407] Connection header received
0619 06:31:05.014987 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.014992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.015075 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.015179 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:05.016724 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.017607 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.019415 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.019859 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.019996 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.023244 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:05.023249 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:05.023252 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:05.023293 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:05.026149 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.026867 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.026872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.026877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.027037 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.027548 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.027553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.027556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.027846 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:05.027952 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.028527 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.028681 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260619 06:31:05.029835 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.010855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:45511)
0619 06:31:05.012505 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.012563 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:05.014724 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:05.015451 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:05.015460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:05.015978 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:31:05.016545 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.016563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.017783 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.017789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.019246 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.019258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.020073 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.020082 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.020307 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.020965 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:31:05.020985 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:31:05.023046 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:31:05.026270 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:05.026278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:05.026292 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:05.026774 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:05.027187 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:05.027190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:05.027192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:05.027419 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:05.027948 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:31:05.027954 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:31:05.028213 (+ 259us) client_negotiation.cc:769] Sending connection context
0619 06:31:05.028451 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 06:31:05.028779 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1360,"thread_start_us":324,"threads_started":1}
I20260619 06:31:05.031180 16372 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40179
I20260619 06:31:05.031524 16372 heartbeater.cc:464] Registering TS with master...
I20260619 06:31:05.032140 16372 heartbeater.cc:511] Master 127.0.97.126:40179 requested a full tablet report, sending...
I20260619 06:31:05.033499 15915 ts_manager.cc:194] Registered new tserver with Master: 18af0be18e764bab9958b7336d6689ba (127.0.97.67:39185)
I20260619 06:31:05.034310 15915 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:45511
I20260619 06:31:05.039544 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:31:05.056427 16107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.044025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48936 (local address 127.0.97.126:40179)
0619 06:31:05.044175 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.044180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.044674 (+ 494us) server_negotiation.cc:407] Connection header received
0619 06:31:05.044819 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.044823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.044878 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.044987 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:05.046355 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.047045 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.048385 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.048769 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.048860 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.051490 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:05.051494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:05.051497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:05.051546 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:05.054252 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.054871 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.054874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.054877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.055016 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.055354 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.055357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.055359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.055651 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:05.055821 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.056056 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.056202 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:31:05.060180 15915 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:31:05.067122 15915 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:31:05.089891 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.080155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35454 (local address 127.0.97.67:39185)
0619 06:31:05.081224 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.081232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.081263 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:31:05.081362 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.081369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.081607 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.082535 (+ 928us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:05.082846 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.084321 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.087391 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.088721 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.088833 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.089053 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.089398 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.089554 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":1276,"threads_started":1}
I20260619 06:31:05.089934 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.079458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39185 (local address 127.0.0.1:35454)
0619 06:31:05.080438 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.080491 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:05.080664 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:05.081794 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:05.081800 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:05.081829 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:05.082335 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.082348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.084518 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.084523 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.087242 (+ 2719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.087255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.089055 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.089092 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.089223 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.089246 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:31:05.089352 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:31:05.089479 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":60,"thread_start_us":1082,"threads_started":1}
I20260619 06:31:05.090056 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.079345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.0.97.65:41525)
0619 06:31:05.080041 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.080051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.080596 (+ 545us) server_negotiation.cc:407] Connection header received
0619 06:31:05.080915 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.080925 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.081232 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.081399 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:05.082480 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.084042 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.087065 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.088507 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.088659 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.088891 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.089588 (+ 697us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.089738 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":358,"threads_started":1}
I20260619 06:31:05.090149 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.081353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.0.97.66:37251)
0619 06:31:05.082020 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.082031 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.082657 (+ 626us) server_negotiation.cc:407] Connection header received
0619 06:31:05.082824 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.082832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.083035 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.083216 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:05.083812 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.085308 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.087645 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.088661 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.088780 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.088969 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.089684 (+ 715us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.089829 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":398,"threads_started":1}
I20260619 06:31:05.090689 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.080848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37251 (local address 127.0.0.1:37032)
0619 06:31:05.082532 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.082550 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:05.082684 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:05.083227 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:05.083232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:05.083257 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:05.083608 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.083620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.085446 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.085452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.087054 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.087068 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.089146 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.089152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.089308 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.089339 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:31:05.089423 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 06:31:05.089555 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1578,"mutex_wait_us":67,"thread_start_us":1433,"threads_started":1}
I20260619 06:31:05.090732 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.079048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42642)
0619 06:31:05.080438 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.080491 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:05.080675 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:05.081708 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:05.081718 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:05.081760 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:05.082261 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.082280 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.084237 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.084243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.086893 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.086907 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.088717 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.088727 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.088913 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.088958 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:31:05.089117 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 06:31:05.089305 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1123,"mutex_wait_us":55,"thread_start_us":968,"threads_started":1}
I20260619 06:31:05.093999 16174 tablet_service.cc:1467] Processing CreateTablet for tablet 83ae7aba2ff648899ab1697da606c16e (DEFAULT_TABLE table=test-table [id=4c78c2b0809b4609b1a433416873f56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:31:05.094132 16307 tablet_service.cc:1467] Processing CreateTablet for tablet 83ae7aba2ff648899ab1697da606c16e (DEFAULT_TABLE table=test-table [id=4c78c2b0809b4609b1a433416873f56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:31:05.094132 16040 tablet_service.cc:1467] Processing CreateTablet for tablet 83ae7aba2ff648899ab1697da606c16e (DEFAULT_TABLE table=test-table [id=4c78c2b0809b4609b1a433416873f56a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:31:05.095221 16174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ae7aba2ff648899ab1697da606c16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:31:05.095464 16307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ae7aba2ff648899ab1697da606c16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:31:05.095458 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ae7aba2ff648899ab1697da606c16e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:31:05.106001 16390 tablet_bootstrap.cc:492] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Bootstrap starting.
I20260619 06:31:05.109201 16391 tablet_bootstrap.cc:492] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Bootstrap starting.
I20260619 06:31:05.110105 16390 tablet_bootstrap.cc:654] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Neither blocks nor log segments found. Creating new log.
I20260619 06:31:05.111485 16390 log.cc:824] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:05.112582 16391 tablet_bootstrap.cc:654] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Neither blocks nor log segments found. Creating new log.
I20260619 06:31:05.112871 16392 tablet_bootstrap.cc:492] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Bootstrap starting.
I20260619 06:31:05.113847 16391 log.cc:824] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:05.114460 16390 tablet_bootstrap.cc:492] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: No bootstrap required, opened a new log
I20260619 06:31:05.114782 16390 ts_tablet_manager.cc:1397] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260619 06:31:05.116168 16392 tablet_bootstrap.cc:654] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Neither blocks nor log segments found. Creating new log.
I20260619 06:31:05.117048 16391 tablet_bootstrap.cc:492] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: No bootstrap required, opened a new log
I20260619 06:31:05.117355 16391 ts_tablet_manager.cc:1397] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:31:05.117429 16392 log.cc:824] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:05.122520 16390 raft_consensus.cc:348] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.122862 16390 raft_consensus.cc:374] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:31:05.122992 16390 raft_consensus.cc:729] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406d763d570246fbaa2549e3144e8a7f, State: Initialized, Role: FOLLOWER
I20260619 06:31:05.123476 16392 tablet_bootstrap.cc:492] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: No bootstrap required, opened a new log
I20260619 06:31:05.123718 16392 ts_tablet_manager.cc:1397] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260619 06:31:05.123593 16390 consensus_queue.cc:260] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.124635 16239 heartbeater.cc:503] Master 127.0.97.126:40179 was elected leader, sending a full tablet report...
I20260619 06:31:05.125185 16390 ts_tablet_manager.cc:1428] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260619 06:31:05.128541 16391 raft_consensus.cc:348] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.128937 16391 raft_consensus.cc:374] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:31:05.129037 16391 raft_consensus.cc:729] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18af0be18e764bab9958b7336d6689ba, State: Initialized, Role: FOLLOWER
I20260619 06:31:05.129595 16391 consensus_queue.cc:260] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.131181 16372 heartbeater.cc:503] Master 127.0.97.126:40179 was elected leader, sending a full tablet report...
I20260619 06:31:05.131860 16391 ts_tablet_manager.cc:1428] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260619 06:31:05.132660 16392 raft_consensus.cc:348] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.133102 16392 raft_consensus.cc:374] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:31:05.133211 16392 raft_consensus.cc:729] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e868380c534359b78cac570b48a3da, State: Initialized, Role: FOLLOWER
I20260619 06:31:05.133774 16392 consensus_queue.cc:260] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.135136 16392 ts_tablet_manager.cc:1428] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Time spent starting tablet: real 0.011s user 0.012s sys 0.001s
W20260619 06:31:05.234035 16106 tablet.cc:2369] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:31:05.248515 16240 tablet.cc:2369] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:31:05.259389 16373 tablet.cc:2369] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:31:05.287431 16397 raft_consensus.cc:482] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:31:05.287869 16397 raft_consensus.cc:504] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.289656 16397 leader_election.cc:288] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7e868380c534359b78cac570b48a3da (127.0.97.65:41525), 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251)
I20260619 06:31:05.296176 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.290110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:59057 (local address 127.0.97.65:41525)
0619 06:31:05.290361 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.290367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.290389 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:31:05.290622 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.290628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.290741 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.290882 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:05.291731 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.292497 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.294338 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.295166 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.295329 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.295443 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.295750 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.295896 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:31:05.296216 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.290639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:49951 (local address 127.0.97.66:37251)
0619 06:31:05.290883 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.290889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.291295 (+ 406us) server_negotiation.cc:407] Connection header received
0619 06:31:05.291421 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.291428 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.291503 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.291580 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:05.292067 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.292754 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.294467 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.295222 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.295382 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.295475 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.295806 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.295935 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:31:05.296240 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.289993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.97.67:59057)
0619 06:31:05.290254 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.290275 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:05.290446 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:05.290987 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:05.290997 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:05.291032 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:05.291236 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.291273 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.292650 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.292658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.294158 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.294172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.295334 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.295341 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.295580 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.295605 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:31:05.295725 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 06:31:05.295894 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":6}
I20260619 06:31:05.296353 16399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.290442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37251 (local address 127.0.97.67:49951)
0619 06:31:05.291177 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.291198 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:05.291300 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:05.291605 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:05.291607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:05.291624 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:05.291902 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.291908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.292920 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.292924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.294315 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:05.294327 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.295431 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.295437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:05.295644 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.295667 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 06:31:05.295771 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:31:05.295932 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":77,"thread_start_us":447,"threads_started":1}
I20260619 06:31:05.297412 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ae7aba2ff648899ab1697da606c16e" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7e868380c534359b78cac570b48a3da" is_pre_election: true
I20260619 06:31:05.297439 16194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ae7aba2ff648899ab1697da606c16e" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406d763d570246fbaa2549e3144e8a7f" is_pre_election: true
I20260619 06:31:05.297902 16060 raft_consensus.cc:2454] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 0.
I20260619 06:31:05.297936 16194 raft_consensus.cc:2454] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 0.
I20260619 06:31:05.298584 16261 leader_election.cc:302] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18af0be18e764bab9958b7336d6689ba, 406d763d570246fbaa2549e3144e8a7f; no voters:
I20260619 06:31:05.298966 16397 raft_consensus.cc:2781] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:31:05.299114 16397 raft_consensus.cc:482] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:31:05.299245 16397 raft_consensus.cc:3037] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:31:05.301419 16397 raft_consensus.cc:504] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.301980 16397 leader_election.cc:288] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 election: Requested vote from peers a7e868380c534359b78cac570b48a3da (127.0.97.65:41525), 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251)
I20260619 06:31:05.302533 16060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ae7aba2ff648899ab1697da606c16e" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7e868380c534359b78cac570b48a3da"
I20260619 06:31:05.302533 16194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83ae7aba2ff648899ab1697da606c16e" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406d763d570246fbaa2549e3144e8a7f"
I20260619 06:31:05.302754 16194 raft_consensus.cc:3037] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:31:05.302754 16060 raft_consensus.cc:3037] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:31:05.305943 16060 raft_consensus.cc:2454] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 1.
I20260619 06:31:05.305953 16194 raft_consensus.cc:2454] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 1.
I20260619 06:31:05.306329 16261 leader_election.cc:302] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18af0be18e764bab9958b7336d6689ba, 406d763d570246fbaa2549e3144e8a7f; no voters:
I20260619 06:31:05.306586 16397 raft_consensus.cc:2781] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:31:05.307021 16397 raft_consensus.cc:686] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 1 LEADER]: Becoming Leader. State: Replica: 18af0be18e764bab9958b7336d6689ba, State: Running, Role: LEADER
I20260619 06:31:05.307730 16397 consensus_queue.cc:237] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } }
I20260619 06:31:05.312287 15916 catalog_manager.cc:5495] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 18af0be18e764bab9958b7336d6689ba (127.0.97.67). New cstate: current_term: 1 leader_uuid: "18af0be18e764bab9958b7336d6689ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } health_report { overall_health: UNKNOWN } } }
I20260619 06:31:05.383144 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.377648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.97.67:39185)
0619 06:31:05.377840 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.377847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.377962 (+ 115us) server_negotiation.cc:407] Connection header received
0619 06:31:05.378232 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.378236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.378319 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.378495 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:31:05.378886 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.379575 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.381016 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.381449 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.381581 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.382067 (+ 486us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:31:05.382564 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:31:05.382742 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.382798 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.382874 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:31:05.391176 16060 raft_consensus.cc:1261] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Refusing update from remote peer 18af0be18e764bab9958b7336d6689ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:31:05.391176 16194 raft_consensus.cc:1261] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Refusing update from remote peer 18af0be18e764bab9958b7336d6689ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:31:05.392129 16400 consensus_queue.cc:1035] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:31:05.392490 16397 consensus_queue.cc:1035] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:31:05.404884 16405 mvcc.cc:205] Tried to move back new op lower bound from 7298460325431795712 to 7298460325103837184. Current Snapshot: MvccSnapshot[applied={T|T < 7298460325431795712}]
I20260619 06:31:05.419458 15915 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:31:05.419776 15915 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48936:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:31:05.423725 15915 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Sending DeleteTablet for 3 replicas of tablet 83ae7aba2ff648899ab1697da606c16e
I20260619 06:31:05.424804 16040 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ae7aba2ff648899ab1697da606c16e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:31:05 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:42642
I20260619 06:31:05.424835 16307 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ae7aba2ff648899ab1697da606c16e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:31:05 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35454
I20260619 06:31:05.426618 16174 tablet_service.cc:1514] Processing DeleteTablet for tablet 83ae7aba2ff648899ab1697da606c16e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:31:05 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:37032
I20260619 06:31:05.427386 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 15977
I20260619 06:31:05.436041 16414 tablet_replica.cc:330] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: stopping tablet replica
I20260619 06:31:05.436476 16414 raft_consensus.cc:2229] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:31:05.436964 16414 raft_consensus.cc:2258] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:31:05.438639 16413 tablet_replica.cc:330] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: stopping tablet replica
I20260619 06:31:05.439033 16413 raft_consensus.cc:2229] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:31:05.439790 16413 raft_consensus.cc:2258] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:31:05.441996 16412 tablet_replica.cc:330] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: stopping tablet replica
I20260619 06:31:05.443228 16414 ts_tablet_manager.cc:1905] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:31:05.444068 16412 raft_consensus.cc:2229] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:31:05.444144 16413 ts_tablet_manager.cc:1905] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:31:05.444568 16412 raft_consensus.cc:2258] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Raft consensus is shut down!
Jun 19 06:31:05 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:31:05.451444 16412 ts_tablet_manager.cc:1905] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:31:05.453094 16413 ts_tablet_manager.cc:1918] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:31:05.453313 16413 log.cc:1196] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/83ae7aba2ff648899ab1697da606c16e
I20260619 06:31:05.453869 16413 ts_tablet_manager.cc:1939] T 83ae7aba2ff648899ab1697da606c16e P 18af0be18e764bab9958b7336d6689ba: Deleting consensus metadata
I20260619 06:31:05.455996 15903 catalog_manager.cc:4841] TS 18af0be18e764bab9958b7336d6689ba (127.0.97.67:39185): tablet 83ae7aba2ff648899ab1697da606c16e (table test-table [id=4c78c2b0809b4609b1a433416873f56a]) successfully deleted
I20260619 06:31:05.457011 16414 ts_tablet_manager.cc:1918] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:31:05.457231 16414 log.cc:1196] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/83ae7aba2ff648899ab1697da606c16e
I20260619 06:31:05.457577 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.428452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42658 (local address 127.0.97.65:41525)
0619 06:31:05.428612 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.428618 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.434817 (+ 6199us) server_negotiation.cc:407] Connection header received
0619 06:31:05.435005 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.435014 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.435145 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.435275 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:05.438487 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.440238 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.443364 (+ 3126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.444057 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.444177 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.449809 (+ 5632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:05.449829 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:05.449858 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:05.449926 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:05.454015 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.454936 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.454945 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.454948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.455129 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.455686 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.455692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.455696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.456156 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:05.456433 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.456909 (+ 476us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.457157 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 06:31:05.457749 16414 ts_tablet_manager.cc:1939] T 83ae7aba2ff648899ab1697da606c16e P 406d763d570246fbaa2549e3144e8a7f: Deleting consensus metadata
I20260619 06:31:05.458473 16100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:05.459426 15901 catalog_manager.cc:4841] TS 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251): tablet 83ae7aba2ff648899ab1697da606c16e (table test-table [id=4c78c2b0809b4609b1a433416873f56a]) successfully deleted
I20260619 06:31:05.566089 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 15977
I20260619 06:31:05.583406 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 16111
W20260619 06:31:05.583595 15901 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:41525 (error 108)
I20260619 06:31:05.587560 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.587018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42674)
0619 06:31:05.587245 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.587391 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 06:31:05.587841 15901 catalog_manager.cc:4584] TS a7e868380c534359b78cac570b48a3da (127.0.97.65:41525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83ae7aba2ff648899ab1697da606c16e: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Jun 19 06:31:05 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:31:05.601715 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.584253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.0.97.66:37251)
0619 06:31:05.584464 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.584471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.584495 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:31:05.584714 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.584722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.584821 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.584935 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:05.586866 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.587925 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.589493 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.590141 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.590298 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.594669 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:05.594691 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:05.594712 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:05.594766 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:05.598626 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.599356 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.599369 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.599374 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.599536 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.599890 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.599895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.599898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.600469 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:05.600779 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.601023 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.601276 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:31:05.602563 16234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:05.641649 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.641114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42680)
0619 06:31:05.641346 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.641454 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:31:05.642884 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.642443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42692)
0619 06:31:05.642618 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.642716 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 06:31:05.693593 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 16111
I20260619 06:31:05.703889 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.703242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42702)
0619 06:31:05.703517 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.703638 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 06:31:05.705238 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.704745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42712)
0619 06:31:05.704932 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.705033 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:31:05.711174 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 16244
Jun 19 06:31:05 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:31:05.727140 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.712029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35480 (local address 127.0.97.67:39185)
0619 06:31:05.712247 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:05.712254 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:05.712273 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:31:05.712348 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:05.712351 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:05.712407 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:05.712527 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:05.713815 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.714657 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.716017 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:05.716486 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:05.716633 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:05.721163 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:05.721174 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:05.721203 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:05.721255 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:05.724233 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.724918 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.724925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.724928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.725231 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:05.725622 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:05.725627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:05.725630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:05.725965 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:05.726269 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:05.726577 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 06:31:05.726806 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:31:05.727819 16367 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:05.785727 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.785130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42716)
0619 06:31:05.785371 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.785496 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260619 06:31:05.786953 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.786443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42724)
0619 06:31:05.786707 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.786798 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
I20260619 06:31:05.850035 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 16244
I20260619 06:31:05.865397 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 15884
I20260619 06:31:05.866596 15946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:05.976737 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:05.976114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:42730)
0619 06:31:05.976377 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:05.976495 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:41525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 06:31:05.976763 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 15884
I20260619 06:31:25.994247 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40179
--webserver_interface=127.0.97.126
--webserver_port=39731
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:31:26.319065 16421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:26.319325 16421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:26.319555 16421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:26.326575 16421 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:31:26.326675 16421 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:31:26.326758 16421 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:31:26.326857 16421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:31:26.326946 16421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:26.327061 16421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:26.327147 16421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:31:26.327221 16421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:31:26.336582 16421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:26.337893 16421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:26.339244 16421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:26.360805 16428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:26.360853 16427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:26.360821 16430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:26.361723 16421 server_base.cc:1034] running on GCE node
I20260619 06:31:26.362485 16421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:26.363899 16421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:26.365170 16421 hybrid_clock.cc:648] HybridClock initialized: now 1781850686365098 us; error 52 us; skew 500 ppm
I20260619 06:31:26.365581 16421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:26 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:26.372642 16421 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:31:26.374305 16421 webserver.cc:460] Webserver started at http://127.0.97.126:39731/ using document root <none> and password file <none>
I20260619 06:31:26.375141 16421 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:26.375279 16421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:26.381155 16421 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 06:31:26.383937 16437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:26.385344 16421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:31:26.385525 16421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "bb19f564665a46c09f121f1b7375cc29"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "4e8b2a3beb1165972e83ec79502a3d58"
server_key_iv: "854989160e0242d5f3e4e7c2d1f21744"
server_key_version: "encryptionkey@0"
I20260619 06:31:26.386565 16421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:26.415634 16421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:26.419806 16421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:26.420094 16421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:26.438632 16421 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:40179
I20260619 06:31:26.438665 16488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:40179 every 8 connection(s)
I20260619 06:31:26.440111 16421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:31:26.446113 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 16421
I20260619 06:31:26.447150 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:41525
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:40179
--webserver_port=35983
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:31:26.447912 16489 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:31:26.453147 16489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Bootstrap starting.
I20260619 06:31:26.473572 16489 log.cc:824] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:26.485404 16489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:31:26.485976 16489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Bootstrap complete.
I20260619 06:31:26.495301 16489 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:26.496176 16489 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb19f564665a46c09f121f1b7375cc29, State: Initialized, Role: FOLLOWER
I20260619 06:31:26.496861 16489 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:26.497112 16489 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:31:26.497233 16489 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:31:26.497419 16489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 1 FOLLOWER]: Advancing to term 2
I20260619 06:31:26.501204 16489 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:26.501752 16489 leader_election.cc:302] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb19f564665a46c09f121f1b7375cc29; no voters:
I20260619 06:31:26.502458 16489 leader_election.cc:288] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 06:31:26.502597 16493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 06:31:26.503831 16493 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 LEADER]: Becoming Leader. State: Replica: bb19f564665a46c09f121f1b7375cc29, State: Running, Role: LEADER
I20260619 06:31:26.504441 16493 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:26.504678 16489 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:31:26.507256 16494 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "bb19f564665a46c09f121f1b7375cc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } } }
I20260619 06:31:26.507565 16494 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:31:26.506870 16495 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb19f564665a46c09f121f1b7375cc29. Latest consensus state: current_term: 2 leader_uuid: "bb19f564665a46c09f121f1b7375cc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } } }
I20260619 06:31:26.508143 16495 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:31:26.519091 16499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:31:26.528949 16499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:31:26.530377 16499 catalog_manager.cc:1259] Loaded cluster ID: 0358a5768b594044b038b5c56d679988
I20260619 06:31:26.530531 16499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:31:26.534832 16499 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:31:26.535784 16510 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:31:26.536880 16499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Loaded TSK: 0
I20260619 06:31:26.567006 16499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Generated new TSK 1
I20260619 06:31:26.567394 16499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:31:26.792266 16491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:26.792531 16491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:26.792694 16491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:26.800122 16491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:26.800223 16491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:26.800429 16491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:31:26.810037 16491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:41525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=35983
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:26.811241 16491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:26.812477 16491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:26.826933 16516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:26.826959 16517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:26.827420 16519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:26.828467 16491 server_base.cc:1034] running on GCE node
I20260619 06:31:26.829190 16491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:26.830336 16491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:26.831563 16491 hybrid_clock.cc:648] HybridClock initialized: now 1781850686831495 us; error 60 us; skew 500 ppm
I20260619 06:31:26.831867 16491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:26 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:26.835044 16491 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:31:26.836562 16491 webserver.cc:460] Webserver started at http://127.0.97.65:35983/ using document root <none> and password file <none>
I20260619 06:31:26.837283 16491 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:26.837435 16491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:26.842556 16491 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260619 06:31:26.846827 16526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:26.848156 16491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:31:26.848309 16491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "a7e868380c534359b78cac570b48a3da"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "fb1d3e03abe9aaec0e5f3e3ab1af78ed"
server_key_iv: "eaaf2c3a7a28182a7f19c9e6f65cf4a6"
server_key_version: "encryptionkey@0"
I20260619 06:31:26.849339 16491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:26.864705 16491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:26.869194 16491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:26.869464 16491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:26.870469 16491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:26.872479 16533 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260619 06:31:26.878041 16533 ts_tablet_manager.cc:1852] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 06:31:26.879081 16533 ts_tablet_manager.cc:1905] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:31:26.886627 16533 ts_tablet_manager.cc:1918] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:31:26.886804 16533 log.cc:1196] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/83ae7aba2ff648899ab1697da606c16e
I20260619 06:31:26.887326 16533 ts_tablet_manager.cc:1939] T 83ae7aba2ff648899ab1697da606c16e P a7e868380c534359b78cac570b48a3da: Deleting consensus metadata
I20260619 06:31:26.887758 16491 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 06:31:26.887866 16491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260619 06:31:26.887967 16491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:26.888031 16491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:26.922995 16491 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41525
I20260619 06:31:26.923072 16639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41525 every 8 connection(s)
I20260619 06:31:26.924336 16491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:31:26.932474 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 16491
I20260619 06:31:26.933215 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:37251
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:40179
--webserver_port=38447
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:31:26 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:31:26.950343 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:26.927538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:43599)
0619 06:31:26.928405 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:26.928457 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:26.930439 (+ 1982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:26.931725 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:26.931732 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:26.932221 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:31:26.932830 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:26.932849 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:26.935305 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:26.935309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:26.936529 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:26.936541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:26.938049 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:26.938053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:26.938231 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:26.938923 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:31:26.938948 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:31:26.941476 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:31:26.946182 (+ 4706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:26.946189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:26.946200 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:26.946647 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:26.947163 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:26.947166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:26.947168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:26.947358 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:26.948300 (+ 942us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:31:26.948306 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:31:26.948621 (+ 315us) client_negotiation.cc:769] Sending connection context
0619 06:31:26.948856 (+ 235us) client_negotiation.cc:240] Negotiation successful
0619 06:31:26.949334 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":364,"threads_started":1}
I20260619 06:31:26.950872 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:26.927277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:43599 (local address 127.0.97.126:40179)
0619 06:31:26.927875 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:26.927887 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:26.928588 (+ 701us) server_negotiation.cc:407] Connection header received
0619 06:31:26.930928 (+ 2340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:26.930941 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:26.931274 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:26.931582 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:26.933028 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:26.935126 (+ 2098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:26.936678 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:26.937847 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:26.938045 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:26.941797 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:26.941810 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:26.941833 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:26.941884 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:26.946029 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:26.946800 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:26.946812 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:26.946822 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:26.947041 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:26.947465 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:26.947472 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:26.947478 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:26.948152 (+ 674us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:26.948353 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:26.949143 (+ 790us) server_negotiation.cc:299] Negotiation successful
0619 06:31:26.949600 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260619 06:31:26.953677 16640 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40179
I20260619 06:31:26.953996 16640 heartbeater.cc:464] Registering TS with master...
I20260619 06:31:26.954660 16640 heartbeater.cc:511] Master 127.0.97.126:40179 requested a full tablet report, sending...
I20260619 06:31:26.956862 16454 ts_manager.cc:194] Registered new tserver with Master: a7e868380c534359b78cac570b48a3da (127.0.97.65:41525)
I20260619 06:31:26.958389 16454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:43599
W20260619 06:31:27.255754 16644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:27.256050 16644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:27.256224 16644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:27.263352 16644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:27.263499 16644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:27.263674 16644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:31:27.274252 16644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:37251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=38447
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:27.275548 16644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:27.276892 16644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:27.290788 16649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:27.290829 16650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:27.290858 16652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:27.291811 16644 server_base.cc:1034] running on GCE node
I20260619 06:31:27.292404 16644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:27.293435 16644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:27.294633 16644 hybrid_clock.cc:648] HybridClock initialized: now 1781850687294594 us; error 29 us; skew 500 ppm
I20260619 06:31:27.295006 16644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:27 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:27.298164 16644 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:31:27.299582 16644 webserver.cc:460] Webserver started at http://127.0.97.66:38447/ using document root <none> and password file <none>
I20260619 06:31:27.300251 16644 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:27.300410 16644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:27.305799 16644 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:31:27.308771 16659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:27.310205 16644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:31:27.310372 16644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "406d763d570246fbaa2549e3144e8a7f"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "afb88a578d9494b26736b8d07e25f4f5"
server_key_iv: "4c647fd2e3c34de10effb31b380258ae"
server_key_version: "encryptionkey@0"
I20260619 06:31:27.311340 16644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:27.327674 16644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:27.331511 16644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:27.331832 16644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:27.332840 16644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:27.334120 16644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:27.334227 16644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:27.334357 16644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:27.334453 16644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:27.367257 16644 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37251
I20260619 06:31:27.367341 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37251 every 8 connection(s)
I20260619 06:31:27.368637 16644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:31:27.370146 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 16644
I20260619 06:31:27.370971 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:39185
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:40179
--webserver_port=42217
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 06:31:27 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:31:27.390921 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.372400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:33879 (local address 127.0.97.126:40179)
0619 06:31:27.372612 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.372619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.372793 (+ 174us) server_negotiation.cc:407] Connection header received
0619 06:31:27.374626 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.374635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.374733 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.374856 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:27.376361 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.377563 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.380062 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.380587 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.380684 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.384840 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:27.384845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:27.384848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:27.384880 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:27.387479 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.388368 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.388379 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.388387 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.388571 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.389009 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.389014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.389017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.389377 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:27.389558 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:27.390335 (+ 777us) server_negotiation.cc:299] Negotiation successful
0619 06:31:27.390575 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 06:31:27.391945 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.371824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:33879)
0619 06:31:27.372610 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:27.372655 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:27.374343 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:27.375048 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:27.375056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:27.375641 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:31:27.376176 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.376192 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.377720 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.377727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.379915 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.379933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.380799 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.380808 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.381167 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.381898 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:31:27.381930 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:31:27.384617 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:31:27.387645 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:27.387654 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:27.387670 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:27.388237 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:27.388682 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:27.388687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:27.388699 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:27.388884 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:27.389505 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:31:27.389512 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:31:27.389866 (+ 354us) client_negotiation.cc:769] Sending connection context
0619 06:31:27.390237 (+ 371us) client_negotiation.cc:240] Negotiation successful
0619 06:31:27.390781 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":342,"threads_started":1}
I20260619 06:31:27.393554 16772 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40179
I20260619 06:31:27.393858 16772 heartbeater.cc:464] Registering TS with master...
I20260619 06:31:27.394563 16772 heartbeater.cc:511] Master 127.0.97.126:40179 requested a full tablet report, sending...
I20260619 06:31:27.396142 16454 ts_manager.cc:194] Registered new tserver with Master: 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251)
I20260619 06:31:27.397302 16454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:33879
W20260619 06:31:27.695078 16775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:27.695331 16775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:27.695421 16775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:27.702369 16775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:27.702459 16775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:27.702565 16775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:31:27.711697 16775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:39185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=42217
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:27.712754 16775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:27.713905 16775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:27.726272 16780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:27.726327 16781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:27.726344 16783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:27.727314 16775 server_base.cc:1034] running on GCE node
I20260619 06:31:27.727819 16775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:27.728657 16775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:27.729867 16775 hybrid_clock.cc:648] HybridClock initialized: now 1781850687729833 us; error 67 us; skew 500 ppm
I20260619 06:31:27.730160 16775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:27 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:27.732969 16775 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:31:27.734138 16775 webserver.cc:460] Webserver started at http://127.0.97.67:42217/ using document root <none> and password file <none>
I20260619 06:31:27.734757 16775 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:27.734910 16775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:27.739831 16775 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260619 06:31:27.742486 16790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:27.743651 16775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 06:31:27.743831 16775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "18af0be18e764bab9958b7336d6689ba"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "895adf0e60cad5e92f2e44d8d8716bcf"
server_key_iv: "845b3d9f3a548ee86cfad4732f01770e"
server_key_version: "encryptionkey@0"
I20260619 06:31:27.744701 16775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:27.767580 16775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:27.771289 16775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:27.771555 16775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:27.772486 16775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:27.773756 16775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:27.773880 16775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:27.774008 16775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:27.774094 16775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:27.806954 16775 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39185
I20260619 06:31:27.807039 16902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39185 every 8 connection(s)
I20260619 06:31:27.808322 16775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:31:27.815181 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 16775
Jun 19 06:31:27 dist-test-slave-4t1f krb5kdc[15871](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:31:27.828173 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.810817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:32969 (local address 127.0.97.126:40179)
0619 06:31:27.811000 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.811004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.812019 (+ 1015us) server_negotiation.cc:407] Connection header received
0619 06:31:27.813786 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.813790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.813855 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.813942 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:27.815670 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.816640 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.818183 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.818622 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.818758 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.822262 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:27.822266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:27.822269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:27.822293 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:27.824939 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.825860 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.825868 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.825873 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.826022 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.826434 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.826439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.826443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.826832 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:27.826988 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:27.827580 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 06:31:27.827821 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:31:27.829041 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.811193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:32969)
0619 06:31:27.811860 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:27.811895 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:27.813553 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:27.814149 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:27.814156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:27.814659 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:31:27.815456 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.815483 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.816819 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.816822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.818040 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.818052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.818850 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.818857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.819073 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.819718 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:31:27.819746 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:31:27.822066 (+ 2320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:31:27.825115 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:27.825123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:27.825142 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:27.825728 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:27.826134 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:31:27.826137 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:31:27.826140 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:31:27.826314 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:31:27.826963 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:31:27.826968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:31:27.827231 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 06:31:27.827437 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 06:31:27.827776 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":257,"threads_started":1}
I20260619 06:31:27.830708 16903 heartbeater.cc:347] Connected to a master server at 127.0.97.126:40179
I20260619 06:31:27.831039 16903 heartbeater.cc:464] Registering TS with master...
I20260619 06:31:27.831041 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.816111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40602 (local address 127.0.97.126:40179)
0619 06:31:27.816696 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.816706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.817257 (+ 551us) server_negotiation.cc:407] Connection header received
0619 06:31:27.817446 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.817450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.817515 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.817621 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:27.819058 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.819783 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.821377 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.821879 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.822031 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.824742 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:27.824749 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:27.824754 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:27.824802 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:27.827923 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.828701 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.828705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.828707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.828808 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.829275 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.829281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.829284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.829590 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:27.829818 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:27.830400 (+ 582us) server_negotiation.cc:299] Negotiation successful
0619 06:31:27.830569 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":264,"threads_started":1}
I20260619 06:31:27.831902 16903 heartbeater.cc:511] Master 127.0.97.126:40179 requested a full tablet report, sending...
I20260619 06:31:27.833532 16454 ts_manager.cc:194] Registered new tserver with Master: 18af0be18e764bab9958b7336d6689ba (127.0.97.67:39185)
I20260619 06:31:27.834316 16454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:32969
I20260619 06:31:27.845520 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:31:27.851456 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.846311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.97.126:40179)
0619 06:31:27.846516 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.846524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.846865 (+ 341us) server_negotiation.cc:407] Connection header received
0619 06:31:27.847040 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.847044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.847153 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.847271 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:31:27.847666 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.848345 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.849876 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.850257 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.850344 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.850759 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:31:27.850930 (+ 171us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 06:31:27.851174 (+ 244us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40618: token expired
Metrics: {"server-negotiator.queue_time_us":65}
W20260619 06:31:27.851572 16906 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40618: token expired
W20260619 06:31:27.852017 16377 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.126:40179)
I20260619 06:31:27.863605 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.852509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.0.97.126:40179)
0619 06:31:27.852682 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.852685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.852699 (+ 14us) server_negotiation.cc:407] Connection header received
0619 06:31:27.852733 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.852736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.852791 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.852863 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:27.854213 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.854873 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.856097 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.856452 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.856536 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.858981 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:27.858985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:27.858988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:27.859014 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:27.861615 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.862133 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.862137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.862139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.862248 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:27.862537 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:27.862540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:27.862541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:27.862850 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:27.863018 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:27.863256 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:31:27.863398 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:31:27.866631 16454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:31:27.873917 16454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:31:27.905474 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.887078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.0.97.66:37251)
0619 06:31:27.895609 (+ 8531us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.895623 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.896422 (+ 799us) server_negotiation.cc:407] Connection header received
0619 06:31:27.896816 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.896830 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.897192 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.897393 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:27.898310 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.899904 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.902596 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.904051 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.904211 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.904410 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:27.904788 (+ 378us) server_negotiation.cc:299] Negotiation successful
0619 06:31:27.904973 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8301,"thread_start_us":8169,"threads_started":1}
I20260619 06:31:27.905944 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.885168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.0.1:39348)
0619 06:31:27.886183 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:27.886232 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:27.886435 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:27.898631 (+ 12196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:27.898638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:27.898668 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:27.899068 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.899080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.900890 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.900895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.903095 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.903107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.904954 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.904961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.905177 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.905228 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 06:31:27.905310 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 06:31:27.905442 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":1,"thread_start_us":570,"threads_started":1}
I20260619 06:31:27.905702 16916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.886686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37251 (local address 127.0.0.1:38214)
0619 06:31:27.896281 (+ 9595us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:27.896305 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:27.896554 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:27.897641 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:27.897653 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:27.897704 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:27.898105 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.898120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.900099 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.900105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.902376 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.902392 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.904297 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.904305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.904545 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.904599 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 06:31:27.904729 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 06:31:27.904913 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9339,"mutex_wait_us":139,"thread_start_us":9153,"threads_started":1}
I20260619 06:31:27.906661 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.885302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39348 (local address 127.0.97.65:41525)
0619 06:31:27.898050 (+ 12748us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.898060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.898091 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:31:27.898192 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.898202 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.898424 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.898637 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:27.899228 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.900754 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.903261 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.904692 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.905810 (+ 1118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.906025 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:27.906096 (+ 71us) server_negotiation.cc:299] Negotiation successful
0619 06:31:27.906276 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12569,"thread_start_us":12442,"threads_started":1}
I20260619 06:31:27.907927 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.885830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39185 (local address 127.0.0.1:40918)
0619 06:31:27.895134 (+ 9304us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:27.895157 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:27.895297 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:27.898222 (+ 2925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:27.898227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:27.898250 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:27.898589 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.898604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.900486 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.900495 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.903502 (+ 3007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:27.903512 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.905377 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.905384 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:27.905629 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.905674 (+ 45us) client_negotiation.cc:769] Sending connection context
0619 06:31:27.905780 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 06:31:27.905906 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9144,"mutex_wait_us":28,"thread_start_us":9007,"threads_started":1}
I20260619 06:31:27.908196 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:27.886532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.0.97.67:39185)
0619 06:31:27.897707 (+ 11175us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:27.897718 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:27.897755 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:31:27.897847 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:27.897854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:27.898049 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:27.898233 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:27.898836 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.900354 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.903632 (+ 3278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:27.904857 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:27.904975 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:27.905219 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:27.906032 (+ 813us) server_negotiation.cc:299] Negotiation successful
0619 06:31:27.906202 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10958,"thread_start_us":10775,"threads_started":1}
I20260619 06:31:27.908620 16575 tablet_service.cc:1467] Processing CreateTablet for tablet 8872c739ef0340a2a2aca71042a7bed1 (DEFAULT_TABLE table=test-table [id=2cad28149bd14842b4b4ea7b08579677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:31:27.909777 16575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8872c739ef0340a2a2aca71042a7bed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:31:27.910621 16707 tablet_service.cc:1467] Processing CreateTablet for tablet 8872c739ef0340a2a2aca71042a7bed1 (DEFAULT_TABLE table=test-table [id=2cad28149bd14842b4b4ea7b08579677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:31:27.911770 16838 tablet_service.cc:1467] Processing CreateTablet for tablet 8872c739ef0340a2a2aca71042a7bed1 (DEFAULT_TABLE table=test-table [id=2cad28149bd14842b4b4ea7b08579677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:31:27.912396 16707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8872c739ef0340a2a2aca71042a7bed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:31:27.913153 16838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8872c739ef0340a2a2aca71042a7bed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:31:27.918485 16919 tablet_bootstrap.cc:492] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Bootstrap starting.
I20260619 06:31:27.921538 16919 tablet_bootstrap.cc:654] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Neither blocks nor log segments found. Creating new log.
I20260619 06:31:27.922787 16919 log.cc:824] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:27.930068 16922 tablet_bootstrap.cc:492] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Bootstrap starting.
I20260619 06:31:27.933631 16921 tablet_bootstrap.cc:492] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Bootstrap starting.
I20260619 06:31:27.933696 16919 tablet_bootstrap.cc:492] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: No bootstrap required, opened a new log
I20260619 06:31:27.933948 16919 ts_tablet_manager.cc:1397] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20260619 06:31:27.937368 16921 tablet_bootstrap.cc:654] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Neither blocks nor log segments found. Creating new log.
I20260619 06:31:27.938514 16921 log.cc:824] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:27.939182 16922 tablet_bootstrap.cc:654] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Neither blocks nor log segments found. Creating new log.
I20260619 06:31:27.940344 16922 log.cc:824] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:27.942250 16919 raft_consensus.cc:348] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:27.942633 16919 raft_consensus.cc:374] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:31:27.942735 16919 raft_consensus.cc:729] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e868380c534359b78cac570b48a3da, State: Initialized, Role: FOLLOWER
I20260619 06:31:27.942935 16921 tablet_bootstrap.cc:492] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: No bootstrap required, opened a new log
I20260619 06:31:27.943176 16921 ts_tablet_manager.cc:1397] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20260619 06:31:27.943326 16919 consensus_queue.cc:260] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:27.944303 16640 heartbeater.cc:503] Master 127.0.97.126:40179 was elected leader, sending a full tablet report...
I20260619 06:31:27.944823 16919 ts_tablet_manager.cc:1428] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260619 06:31:27.946765 16922 tablet_bootstrap.cc:492] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: No bootstrap required, opened a new log
I20260619 06:31:27.947036 16922 ts_tablet_manager.cc:1397] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260619 06:31:27.954582 16922 raft_consensus.cc:348] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:27.954919 16922 raft_consensus.cc:374] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:31:27.955018 16922 raft_consensus.cc:729] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18af0be18e764bab9958b7336d6689ba, State: Initialized, Role: FOLLOWER
I20260619 06:31:27.955503 16922 consensus_queue.cc:260] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:27.956470 16903 heartbeater.cc:503] Master 127.0.97.126:40179 was elected leader, sending a full tablet report...
I20260619 06:31:27.956996 16922 ts_tablet_manager.cc:1428] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260619 06:31:27.958108 16921 raft_consensus.cc:348] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:27.958585 16921 raft_consensus.cc:374] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:31:27.958691 16921 raft_consensus.cc:729] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406d763d570246fbaa2549e3144e8a7f, State: Initialized, Role: FOLLOWER
I20260619 06:31:27.959256 16921 consensus_queue.cc:260] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:27.960322 16772 heartbeater.cc:503] Master 127.0.97.126:40179 was elected leader, sending a full tablet report...
I20260619 06:31:27.960754 16921 ts_tablet_manager.cc:1428] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Time spent starting tablet: real 0.017s user 0.008s sys 0.004s
I20260619 06:31:28.012020 16926 raft_consensus.cc:482] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:31:28.012324 16926 raft_consensus.cc:504] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:28.013974 16926 leader_election.cc:288] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7e868380c534359b78cac570b48a3da (127.0.97.65:41525), 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251)
I20260619 06:31:28.021298 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.014315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41525 (local address 127.0.97.67:55201)
0619 06:31:28.014505 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:28.014524 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:28.014653 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:28.015380 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:28.015385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:28.015412 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:28.015580 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:28.015588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.016996 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.017004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:28.019209 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:28.019224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.020576 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.020580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:28.020764 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.020789 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:31:28.020861 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 06:31:28.020948 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":32}
I20260619 06:31:28.021337 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.014285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:55201 (local address 127.0.97.65:41525)
0619 06:31:28.014482 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:28.014486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:28.014593 (+ 107us) server_negotiation.cc:407] Connection header received
0619 06:31:28.014796 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:28.014802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:28.014892 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:28.015027 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:28.015736 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.016874 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.019378 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.020445 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.020575 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.020677 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:28.020921 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 06:31:28.021087 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:31:28.022065 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.015018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:44123 (local address 127.0.97.66:37251)
0619 06:31:28.015282 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:28.015286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:28.015543 (+ 257us) server_negotiation.cc:407] Connection header received
0619 06:31:28.015798 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:28.015802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:28.015858 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:28.015926 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:31:28.016491 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.017454 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.019726 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.020933 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.021057 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.021176 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:28.021609 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 06:31:28.021813 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260619 06:31:28.022200 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8872c739ef0340a2a2aca71042a7bed1" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7e868380c534359b78cac570b48a3da" is_pre_election: true
I20260619 06:31:28.022677 16928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.014740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37251 (local address 127.0.97.67:44123)
0619 06:31:28.015438 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:28.015452 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:31:28.015600 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:31:28.015968 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:31:28.015973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:31:28.015999 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:31:28.016335 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:28.016347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.017587 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.017595 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:28.019596 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:31:28.019606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.021194 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.021199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:31:28.021423 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.021460 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 06:31:28.022264 (+ 804us) client_negotiation.cc:240] Negotiation successful
0619 06:31:28.022363 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":2,"thread_start_us":402,"threads_started":1}
I20260619 06:31:28.022718 16595 raft_consensus.cc:2454] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 0.
I20260619 06:31:28.023452 16792 leader_election.cc:302] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18af0be18e764bab9958b7336d6689ba, a7e868380c534359b78cac570b48a3da; no voters:
I20260619 06:31:28.023604 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8872c739ef0340a2a2aca71042a7bed1" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406d763d570246fbaa2549e3144e8a7f" is_pre_election: true
I20260619 06:31:28.023862 16926 raft_consensus.cc:2781] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:31:28.024144 16926 raft_consensus.cc:482] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:31:28.024113 16727 raft_consensus.cc:2454] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 0.
I20260619 06:31:28.024240 16926 raft_consensus.cc:3037] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:31:28.026557 16926 raft_consensus.cc:504] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:28.027113 16926 leader_election.cc:288] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 election: Requested vote from peers a7e868380c534359b78cac570b48a3da (127.0.97.65:41525), 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251)
I20260619 06:31:28.027493 16595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8872c739ef0340a2a2aca71042a7bed1" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7e868380c534359b78cac570b48a3da"
I20260619 06:31:28.027571 16727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8872c739ef0340a2a2aca71042a7bed1" candidate_uuid: "18af0be18e764bab9958b7336d6689ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406d763d570246fbaa2549e3144e8a7f"
I20260619 06:31:28.027665 16595 raft_consensus.cc:3037] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:31:28.027788 16727 raft_consensus.cc:3037] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:31:28.029848 16595 raft_consensus.cc:2454] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 1.
I20260619 06:31:28.030005 16727 raft_consensus.cc:2454] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18af0be18e764bab9958b7336d6689ba in term 1.
I20260619 06:31:28.030213 16792 leader_election.cc:302] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18af0be18e764bab9958b7336d6689ba, a7e868380c534359b78cac570b48a3da; no voters:
I20260619 06:31:28.030490 16926 raft_consensus.cc:2781] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:31:28.030922 16926 raft_consensus.cc:686] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 1 LEADER]: Becoming Leader. State: Replica: 18af0be18e764bab9958b7336d6689ba, State: Running, Role: LEADER
I20260619 06:31:28.031348 16926 consensus_queue.cc:237] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } }
I20260619 06:31:28.035125 16453 catalog_manager.cc:5495] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 18af0be18e764bab9958b7336d6689ba (127.0.97.67). New cstate: current_term: 1 leader_uuid: "18af0be18e764bab9958b7336d6689ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18af0be18e764bab9958b7336d6689ba" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39185 } health_report { overall_health: HEALTHY } } }
I20260619 06:31:28.059206 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.053599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.97.67:39185)
0619 06:31:28.053825 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:28.053832 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:28.053946 (+ 114us) server_negotiation.cc:407] Connection header received
0619 06:31:28.054194 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:28.054202 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:28.054301 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:28.054450 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:31:28.054828 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.055556 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.056987 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.057461 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.057590 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.058101 (+ 511us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:31:28.058593 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:31:28.058752 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:28.058872 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 06:31:28.058950 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 06:31:28.060009 16904 tablet.cc:2369] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:31:28.067675 16595 raft_consensus.cc:1261] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Refusing update from remote peer 18af0be18e764bab9958b7336d6689ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:31:28.067670 16727 raft_consensus.cc:1261] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Refusing update from remote peer 18af0be18e764bab9958b7336d6689ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:31:28.068663 16926 consensus_queue.cc:1035] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7e868380c534359b78cac570b48a3da" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:31:28.068953 16929 consensus_queue.cc:1035] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "406d763d570246fbaa2549e3144e8a7f" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:31:28.081121 16931 mvcc.cc:205] Tried to move back new op lower bound from 7298460418314280960 to 7298460418179846144. Current Snapshot: MvccSnapshot[applied={T|T < 7298460418314280960}]
I20260619 06:31:28.081634 16932 mvcc.cc:205] Tried to move back new op lower bound from 7298460418314280960 to 7298460418179846144. Current Snapshot: MvccSnapshot[applied={T|T < 7298460418314280960}]
I20260619 06:31:28.097390 16453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:31:28.097684 16453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:31:28.100831 16453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Sending DeleteTablet for 3 replicas of tablet 8872c739ef0340a2a2aca71042a7bed1
I20260619 06:31:28.102160 16838 tablet_service.cc:1514] Processing DeleteTablet for tablet 8872c739ef0340a2a2aca71042a7bed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:31:28 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40918
I20260619 06:31:28.102141 16707 tablet_service.cc:1514] Processing DeleteTablet for tablet 8872c739ef0340a2a2aca71042a7bed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:31:28 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:38214
I20260619 06:31:28.102298 16575 tablet_service.cc:1514] Processing DeleteTablet for tablet 8872c739ef0340a2a2aca71042a7bed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:31:28 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:39348
I20260619 06:31:28.103091 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 16491
I20260619 06:31:28.117957 16943 tablet_replica.cc:330] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: stopping tablet replica
I20260619 06:31:28.118556 16943 raft_consensus.cc:2229] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:31:28.118673 16942 tablet_replica.cc:330] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: stopping tablet replica
I20260619 06:31:28.119076 16942 raft_consensus.cc:2229] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:31:28.119505 16943 raft_consensus.cc:2258] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:31:28.119755 16942 raft_consensus.cc:2258] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:31:28.120543 16941 tablet_replica.cc:330] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: stopping tablet replica
I20260619 06:31:28.121594 16941 raft_consensus.cc:2229] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:31:28.122186 16941 raft_consensus.cc:2258] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:31:28.122305 16942 ts_tablet_manager.cc:1905] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:31:28.122366 16943 ts_tablet_manager.cc:1905] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:31:28.125167 16941 ts_tablet_manager.cc:1905] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:31:28.129537 16942 ts_tablet_manager.cc:1918] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:31:28.129769 16942 log.cc:1196] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/8872c739ef0340a2a2aca71042a7bed1
I20260619 06:31:28.129926 16943 ts_tablet_manager.cc:1918] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:31:28.130154 16943 log.cc:1196] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/8872c739ef0340a2a2aca71042a7bed1
I20260619 06:31:28.130343 16942 ts_tablet_manager.cc:1939] T 8872c739ef0340a2a2aca71042a7bed1 P 18af0be18e764bab9958b7336d6689ba: Deleting consensus metadata
I20260619 06:31:28.130695 16943 ts_tablet_manager.cc:1939] T 8872c739ef0340a2a2aca71042a7bed1 P a7e868380c534359b78cac570b48a3da: Deleting consensus metadata
I20260619 06:31:28.132167 16441 catalog_manager.cc:4841] TS 18af0be18e764bab9958b7336d6689ba (127.0.97.67:39185): tablet 8872c739ef0340a2a2aca71042a7bed1 (table test-table [id=2cad28149bd14842b4b4ea7b08579677]) successfully deleted
I20260619 06:31:28.133255 16439 catalog_manager.cc:4841] TS a7e868380c534359b78cac570b48a3da (127.0.97.65:41525): tablet 8872c739ef0340a2a2aca71042a7bed1 (table test-table [id=2cad28149bd14842b4b4ea7b08579677]) successfully deleted
I20260619 06:31:28.137646 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.105017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.0.97.65:41525)
0619 06:31:28.105289 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:28.105298 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:28.105323 (+ 25us) server_negotiation.cc:407] Connection header received
0619 06:31:28.105417 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:28.105427 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:28.105533 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:28.105695 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:28.108594 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.110239 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.113097 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.114096 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.114285 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.125437 (+ 11152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:28.125450 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:28.125466 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:28.125513 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:28.129821 (+ 4308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:28.131045 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:28.131054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:28.131058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:28.131221 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:28.131735 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:28.131741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:28.131744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:28.132229 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:28.132566 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:28.137241 (+ 4675us) server_negotiation.cc:299] Negotiation successful
0619 06:31:28.137403 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260619 06:31:28.137655 16941 ts_tablet_manager.cc:1918] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:31:28.138159 16941 log.cc:1196] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/8872c739ef0340a2a2aca71042a7bed1
I20260619 06:31:28.138362 16635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:28.138826 16941 ts_tablet_manager.cc:1939] T 8872c739ef0340a2a2aca71042a7bed1 P 406d763d570246fbaa2549e3144e8a7f: Deleting consensus metadata
I20260619 06:31:28.142060 16439 catalog_manager.cc:4841] TS 406d763d570246fbaa2549e3144e8a7f (127.0.97.66:37251): tablet 8872c739ef0340a2a2aca71042a7bed1 (table test-table [id=2cad28149bd14842b4b4ea7b08579677]) successfully deleted
I20260619 06:31:28.260780 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 16491
I20260619 06:31:28.278199 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 16644
I20260619 06:31:28.297211 16918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.279010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.0.97.66:37251)
0619 06:31:28.279208 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:28.279214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:28.279235 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:31:28.279373 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:28.279380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:28.279465 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:28.279566 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:28.280831 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.281969 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.283565 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.284230 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.284349 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.290791 (+ 6442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:28.290811 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:28.290832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:28.290874 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:28.294540 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:28.295150 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:28.295161 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:28.295165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:28.295311 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:28.295640 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:28.295645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:28.295648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:28.296083 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:28.296392 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:28.296588 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 06:31:28.296820 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:31:28.297961 16767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:28.389305 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 16644
I20260619 06:31:28.406731 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 16775
I20260619 06:31:28.423563 16917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:28.407571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40934 (local address 127.0.97.67:39185)
0619 06:31:28.407834 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:31:28.407840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:31:28.407868 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:31:28.407954 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:31:28.407960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:31:28.408052 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:31:28.408170 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:31:28.409593 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.410822 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.412590 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:31:28.413373 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:31:28.413500 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:31:28.417290 (+ 3790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:31:28.417304 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:31:28.417322 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:31:28.417373 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:31:28.420450 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:28.421383 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:28.421392 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:28.421396 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:28.421567 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:31:28.422069 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:31:28.422073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:31:28.422075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:31:28.422398 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:31:28.422632 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 06:31:28.423026 (+ 394us) server_negotiation.cc:299] Negotiation successful
0619 06:31:28.423188 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260619 06:31:28.424302 16898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:28.546058 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 16775
I20260619 06:31:28.566846 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 16421
I20260619 06:31:28.567996 16484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:31:28.678045 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 16421
I20260619 06:31:48.699537 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40179
--webserver_interface=127.0.97.126
--webserver_port=39731
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:31:49.028555 16948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:49.028859 16948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:49.029134 16948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:49.036550 16948 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 06:31:49.036644 16948 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 06:31:49.036741 16948 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 06:31:49.036861 16948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:31:49.036972 16948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:49.037115 16948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:49.037217 16948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:31:49.037304 16948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:31:49.047366 16948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:49.048748 16948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:49.050117 16948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:49.062071 16956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:49.062253 16953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:49.062249 16954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:49.063108 16948 server_base.cc:1034] running on GCE node
I20260619 06:31:49.063767 16948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:49.065042 16948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:49.066319 16948 hybrid_clock.cc:648] HybridClock initialized: now 1781850709066257 us; error 43 us; skew 500 ppm
I20260619 06:31:49.066654 16948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:49 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:49.069777 16948 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:31:49.071169 16948 webserver.cc:460] Webserver started at http://127.0.97.126:39731/ using document root <none> and password file <none>
I20260619 06:31:49.071834 16948 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:49.071975 16948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:49.077382 16948 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:31:49.079990 16963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:49.081351 16948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:31:49.081524 16948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "bb19f564665a46c09f121f1b7375cc29"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "4e8b2a3beb1165972e83ec79502a3d58"
server_key_iv: "854989160e0242d5f3e4e7c2d1f21744"
server_key_version: "encryptionkey@0"
I20260619 06:31:49.082441 16948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:49.103991 16948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:49.107594 16948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:49.107878 16948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:49.126134 16948 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:40179
I20260619 06:31:49.126207 17014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:40179 every 8 connection(s)
I20260619 06:31:49.127538 16948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:31:49.131636 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 16948
I20260619 06:31:49.132647 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:41525
--local_ip_for_outbound_sockets=127.0.97.65
--tserver_master_addrs=127.0.97.126:40179
--webserver_port=35983
--webserver_interface=127.0.97.65
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:31:49.135362 17015 sys_catalog.cc:263] Verifying existing consensus state
I20260619 06:31:49.140367 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Bootstrap starting.
I20260619 06:31:49.160657 17015 log.cc:824] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Log is configured to *not* fsync() on all Append() calls
I20260619 06:31:49.176141 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 06:31:49.176651 17015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Bootstrap complete.
I20260619 06:31:49.185669 17015 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:49.186645 17015 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb19f564665a46c09f121f1b7375cc29, State: Initialized, Role: FOLLOWER
I20260619 06:31:49.187325 17015 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:49.187565 17015 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:31:49.187685 17015 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:31:49.187877 17015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 2 FOLLOWER]: Advancing to term 3
I20260619 06:31:49.191643 17015 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:49.192095 17015 leader_election.cc:302] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb19f564665a46c09f121f1b7375cc29; no voters:
I20260619 06:31:49.192691 17015 leader_election.cc:288] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 06:31:49.192893 17019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 06:31:49.193993 17019 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [term 3 LEADER]: Becoming Leader. State: Replica: bb19f564665a46c09f121f1b7375cc29, State: Running, Role: LEADER
I20260619 06:31:49.194581 17015 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:31:49.194563 17019 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } }
I20260619 06:31:49.197131 17021 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb19f564665a46c09f121f1b7375cc29. Latest consensus state: current_term: 3 leader_uuid: "bb19f564665a46c09f121f1b7375cc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } } }
I20260619 06:31:49.197577 17021 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:31:49.198167 17020 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bb19f564665a46c09f121f1b7375cc29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb19f564665a46c09f121f1b7375cc29" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 40179 } } }
I20260619 06:31:49.198539 17020 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29 [sys.catalog]: This master's current role is: LEADER
I20260619 06:31:49.206207 17026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:31:49.214550 17026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:31:49.216338 17026 catalog_manager.cc:1259] Loaded cluster ID: 0358a5768b594044b038b5c56d679988
I20260619 06:31:49.216480 17026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:31:49.220710 17026 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:31:49.222334 17026 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Loaded TSK: 0
I20260619 06:31:49.222537 17026 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bb19f564665a46c09f121f1b7375cc29: Loaded TSK: 1
F20260619 06:31:49.223665 17026 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1cc05da9ad google::LogMessage::Fail() at ??:0
@ 0x7f1cc05dd88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1cc05da4de google::LogMessage::Flush() at ??:0
@ 0x7f1cc05dba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1cc433e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1cc432d73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1cd60091c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1cd6008783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
I20260619 06:31:49.241586 17037 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f1cd60942a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f1cd60940c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f1cd60de5e9 std::function<>::operator()() at ??:0
@ 0x7f1cd600ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f1cd600ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1cd6092610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f1cd6092439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1cc3226345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1cc322991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f1cc3229799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f1cc3204eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1cc64de6db start_thread at ??:0
@ 0x7f1cbf47461f clone at ??:0
W20260619 06:31:49.515506 17017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:49.515761 17017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:49.515920 17017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:49.523000 17017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:49.523090 17017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:49.523264 17017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:31:49.532558 17017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:41525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=35983
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:49.533880 17017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:49.535110 17017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:49.547940 17062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:49.547824 17063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:49.547910 17065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:49.548619 17017 server_base.cc:1034] running on GCE node
I20260619 06:31:49.549278 17017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:49.550132 17017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:49.551345 17017 hybrid_clock.cc:648] HybridClock initialized: now 1781850709551309 us; error 54 us; skew 500 ppm
I20260619 06:31:49.551641 17017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:49 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850709, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:49.554638 17017 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:31:49.555835 17017 webserver.cc:460] Webserver started at http://127.0.97.65:35983/ using document root <none> and password file <none>
I20260619 06:31:49.556416 17017 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:49.556532 17017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:49.561519 17017 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 06:31:49.563953 17072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:49.565196 17017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:31:49.565321 17017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "a7e868380c534359b78cac570b48a3da"
format_stamp: "Formatted at 2026-06-19 06:31:03 on dist-test-slave-4t1f"
server_key: "fb1d3e03abe9aaec0e5f3e3ab1af78ed"
server_key_iv: "eaaf2c3a7a28182a7f19c9e6f65cf4a6"
server_key_version: "encryptionkey@0"
I20260619 06:31:49.566218 17017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:49.583344 17017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:49.586922 17017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:49.587126 17017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:49.587981 17017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:49.589255 17017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:49.589340 17017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:49.589416 17017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:49.589469 17017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:49.622696 17017 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41525
I20260619 06:31:49.622787 17184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41525 every 8 connection(s)
I20260619 06:31:49.624150 17017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:31:49.628751 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:49.627000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:46587)
0619 06:31:49.627742 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:49.627947 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":296,"threads_started":1}
W20260619 06:31:49.629431 17185 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:40179 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:40179: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
I20260619 06:31:49.629640 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 17017
I20260619 06:31:49.630256 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:49.629958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:46759)
0619 06:31:49.630084 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:49.630164 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260619 06:31:49.630385 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:37251
--local_ip_for_outbound_sockets=127.0.97.66
--tserver_master_addrs=127.0.97.126:40179
--webserver_port=38447
--webserver_interface=127.0.97.66
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:31:49.631115 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:49.630826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:59637)
0619 06:31:49.630949 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:49.631016 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:31:49.631392 17185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 06:31:49.962365 17188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:49.962666 17188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:49.962837 17188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:49.969926 17188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:49.970062 17188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:49.970227 17188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:31:49.979511 17188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:37251
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=38447
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:49.980717 17188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:49.982023 17188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:49.994741 17193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:49.994813 17196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:49.994853 17194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:49.995683 17188 server_base.cc:1034] running on GCE node
I20260619 06:31:49.996347 17188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:49.997370 17188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:49.998639 17188 hybrid_clock.cc:648] HybridClock initialized: now 1781850709998590 us; error 31 us; skew 500 ppm
I20260619 06:31:49.999083 17188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:50 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:50.002566 17188 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:31:50.004139 17188 webserver.cc:460] Webserver started at http://127.0.97.66:38447/ using document root <none> and password file <none>
I20260619 06:31:50.004848 17188 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:50.004982 17188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:50.010430 17188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260619 06:31:50.013168 17203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:50.014559 17188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:31:50.014730 17188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "406d763d570246fbaa2549e3144e8a7f"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "afb88a578d9494b26736b8d07e25f4f5"
server_key_iv: "4c647fd2e3c34de10effb31b380258ae"
server_key_version: "encryptionkey@0"
I20260619 06:31:50.015733 17188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:50.034948 17188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:50.039287 17188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:50.039577 17188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:50.040576 17188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:50.042078 17188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:50.042189 17188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:50.042321 17188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:50.042423 17188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:50.084874 17188 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37251
I20260619 06:31:50.084920 17315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37251 every 8 connection(s)
I20260619 06:31:50.086724 17188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:31:50.092130 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:50.090399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:44987)
0619 06:31:50.091131 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:50.091311 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":294,"threads_started":1}
W20260619 06:31:50.092772 17316 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:40179 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:40179: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
I20260619 06:31:50.093629 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:50.093304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:33191)
0619 06:31:50.093459 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:50.093532 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260619 06:31:50.094431 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:50.094099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:55339)
0619 06:31:50.094247 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:50.094327 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 06:31:50.094449 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 17188
W20260619 06:31:50.094692 17316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:31:50.095108 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:39185
--local_ip_for_outbound_sockets=127.0.97.67
--tserver_master_addrs=127.0.97.126:40179
--webserver_port=42217
--webserver_interface=127.0.97.67
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:31:50.412086 17319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:31:50.412357 17319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:31:50.412451 17319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:31:50.419947 17319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:31:50.420042 17319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:31:50.420150 17319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:31:50.430059 17319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:39185
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=42217
--tserver_master_addrs=127.0.97.126:40179
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:31:50.431237 17319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:31:50.432559 17319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:31:50.445986 17324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:50.446110 17325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:31:50.446110 17327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:31:50.446730 17319 server_base.cc:1034] running on GCE node
I20260619 06:31:50.447341 17319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:31:50.448269 17319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:31:50.449489 17319 hybrid_clock.cc:648] HybridClock initialized: now 1781850710449454 us; error 50 us; skew 500 ppm
I20260619 06:31:50.449792 17319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:31:50 dist-test-slave-4t1f krb5kdc[15871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850710, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:31:50.452914 17319 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:31:50.454280 17319 webserver.cc:460] Webserver started at http://127.0.97.67:42217/ using document root <none> and password file <none>
I20260619 06:31:50.454860 17319 fs_manager.cc:359] Metadata directory not provided
I20260619 06:31:50.454962 17319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:31:50.459955 17319 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260619 06:31:50.462628 17334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:50.463888 17319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:31:50.464152 17319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "18af0be18e764bab9958b7336d6689ba"
format_stamp: "Formatted at 2026-06-19 06:31:04 on dist-test-slave-4t1f"
server_key: "895adf0e60cad5e92f2e44d8d8716bcf"
server_key_iv: "845b3d9f3a548ee86cfad4732f01770e"
server_key_version: "encryptionkey@0"
I20260619 06:31:50.465131 17319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:31:50.494696 17319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:31:50.498745 17319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:31:50.499019 17319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:31:50.500005 17319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:31:50.501356 17319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:31:50.501499 17319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:50.501619 17319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:31:50.501703 17319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:31:50.534826 17319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39185
I20260619 06:31:50.534911 17446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39185 every 8 connection(s)
I20260619 06:31:50.536288 17319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:31:50.536965 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 17319
I20260619 06:31:50.541728 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:50.539862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:48201)
0619 06:31:50.540542 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:50.540743 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":266,"threads_started":1}
W20260619 06:31:50.542434 17447 heartbeater.cc:644] Failed to heartbeat to 127.0.97.126:40179 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.126:40179: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
I20260619 06:31:50.543202 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:50.542891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:54139)
0619 06:31:50.543035 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:50.543102 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260619 06:31:50.544025 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:50.543734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:48729)
0619 06:31:50.543846 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:50.543920 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 06:31:50.544277 17447 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 06:31:50.632658 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:50.631857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:45513)
0619 06:31:50.632302 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:50.632422 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":206,"threads_started":1}
I20260619 06:31:51.096009 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:51.095189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:59465)
0619 06:31:51.095682 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:51.095777 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":269,"threads_started":1}
I20260619 06:31:51.545691 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:51.544829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:45887)
0619 06:31:51.545393 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:51.545473 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":302,"threads_started":1}
I20260619 06:31:51.634291 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:51.633557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:52683)
0619 06:31:51.634060 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:51.634135 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":274,"threads_started":1}
I20260619 06:31:52.097743 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:52.096993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:54711)
0619 06:31:52.097514 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:52.097592 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":275,"threads_started":1}
I20260619 06:31:52.547406 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:52.546563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:52257)
0619 06:31:52.547050 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:52.547192 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":269,"threads_started":1}
I20260619 06:31:52.635927 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:52.635132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:43957)
0619 06:31:52.635639 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:52.635762 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":297,"threads_started":1}
I20260619 06:31:53.099287 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:53.098589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:60203)
0619 06:31:53.099036 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:53.099117 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":235,"threads_started":1}
I20260619 06:31:53.548804 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:53.548166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:48221)
0619 06:31:53.548565 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:53.548638 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":210,"threads_started":1}
I20260619 06:31:53.637496 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:53.636782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:40803)
0619 06:31:53.637223 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:53.637297 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":184,"threads_started":1}
I20260619 06:31:54.100894 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:54.100189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:51003)
0619 06:31:54.100662 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:54.100737 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":270,"threads_started":1}
I20260619 06:31:54.550474 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:54.549752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:60141)
0619 06:31:54.550204 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:54.550297 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":223,"threads_started":1}
I20260619 06:31:54.639057 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:54.638355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:32795)
0619 06:31:54.638802 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:54.638886 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":226,"threads_started":1}
I20260619 06:31:55.102494 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:55.101802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:59365)
0619 06:31:55.102219 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:55.102325 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":235,"threads_started":1}
I20260619 06:31:55.552309 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:55.551578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:40983)
0619 06:31:55.552048 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:55.552130 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":240,"threads_started":1}
I20260619 06:31:55.640729 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:55.639999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:58655)
0619 06:31:55.640474 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:55.640567 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260619 06:31:56.104033 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:56.103275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:45691)
0619 06:31:56.103796 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:56.103889 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":303,"threads_started":1}
I20260619 06:31:56.554183 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:56.553303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:37091)
0619 06:31:56.553800 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:56.553952 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":275,"threads_started":1}
I20260619 06:31:56.642834 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:56.641729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:51069)
0619 06:31:56.642297 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:56.642430 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":273,"threads_started":1}
I20260619 06:31:57.105830 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:57.105008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:42769)
0619 06:31:57.105539 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:57.105615 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":199,"threads_started":1}
I20260619 06:31:57.555936 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:57.555097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:60689)
0619 06:31:57.555601 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:57.555677 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":212,"threads_started":1}
I20260619 06:31:57.644347 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:57.643716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:52321)
0619 06:31:57.644124 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:57.644198 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":218,"threads_started":1}
I20260619 06:31:58.107372 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:58.106695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:55743)
0619 06:31:58.107129 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:58.107214 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":233,"threads_started":1}
I20260619 06:31:58.557453 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:58.556773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:52965)
0619 06:31:58.557215 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:58.557289 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":206,"threads_started":1}
I20260619 06:31:58.645833 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:58.645131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:34817)
0619 06:31:58.645558 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:58.645653 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":202,"threads_started":1}
I20260619 06:31:59.108842 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:59.108172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:49973)
0619 06:31:59.108603 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:59.108681 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":219,"threads_started":1}
I20260619 06:31:59.558936 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:59.558226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:32853)
0619 06:31:59.558685 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:59.558763 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
I20260619 06:31:59.647480 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:31:59.646731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:55039)
0619 06:31:59.647249 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 06:31:59.647326 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":330,"threads_started":1}
I20260619 06:32:00.110421 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:00.109754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:51951)
0619 06:32:00.110163 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:00.110241 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":222,"threads_started":1}
I20260619 06:32:00.560472 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:00.559800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:56303)
0619 06:32:00.560223 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:00.560309 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":236,"threads_started":1}
I20260619 06:32:00.649019 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:00.648351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:44839)
0619 06:32:00.648764 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:00.648847 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":204,"threads_started":1}
I20260619 06:32:01.112061 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:01.111410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:43771)
0619 06:32:01.111802 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:01.111889 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":200,"threads_started":1}
I20260619 06:32:01.561966 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:01.561301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:39001)
0619 06:32:01.561713 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:01.561798 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260619 06:32:01.650692 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:01.650005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:43693)
0619 06:32:01.650460 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:01.650538 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":220,"threads_started":1}
I20260619 06:32:02.113695 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:02.113040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:58889)
0619 06:32:02.113464 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:02.113540 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":212,"threads_started":1}
I20260619 06:32:02.563521 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:02.562858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:36805)
0619 06:32:02.563286 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:02.563363 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260619 06:32:02.652443 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:02.651675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:41033)
0619 06:32:02.652166 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:02.652255 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260619 06:32:03.115238 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:03.114539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:32785)
0619 06:32:03.114973 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:03.115076 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":207,"threads_started":1}
I20260619 06:32:03.565304 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:03.564458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:58375)
0619 06:32:03.564885 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:03.565016 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":219,"threads_started":1}
I20260619 06:32:03.653955 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:03.653341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:34279)
0619 06:32:03.653718 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:03.653790 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":195,"threads_started":1}
I20260619 06:32:04.116922 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:04.116132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:35389)
0619 06:32:04.116660 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:04.116732 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":295,"threads_started":1}
I20260619 06:32:04.566785 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:04.566120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:44517)
0619 06:32:04.566539 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:04.566611 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":217,"threads_started":1}
I20260619 06:32:04.655609 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:04.654944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.65:42785)
0619 06:32:04.655344 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:04.655437 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":213,"threads_started":1}
I20260619 06:32:05.118701 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:05.117919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.66:52475)
0619 06:32:05.118423 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:05.118515 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260619 06:32:05.540768 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 17017
I20260619 06:32:05.562543 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:05.541848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53778 (local address 127.0.97.65:41525)
0619 06:32:05.542469 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:05.542487 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:05.542536 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:32:05.544657 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:05.544670 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:05.545224 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:05.545617 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:05.547329 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:05.548577 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:05.549891 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:05.550523 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:05.550710 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:05.556663 (+ 5953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:05.556671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:05.556685 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:05.556725 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:05.559424 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:05.560085 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:05.560092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:05.560097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:05.560261 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:05.560623 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:05.560629 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:05.560632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:05.561175 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:05.561476 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:05.561855 (+ 379us) server_negotiation.cc:299] Negotiation successful
0619 06:32:05.562256 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":532,"threads_started":1}
I20260619 06:32:05.563862 17180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:05.568881 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:05.567845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:40179 (local address 127.0.97.67:47419)
0619 06:32:05.568431 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:05.568600 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.126:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":254,"threads_started":1}
I20260619 06:32:05.648588 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 17017
I20260619 06:32:05.665558 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 17188
I20260619 06:32:05.682222 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:05.666454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.0.97.66:37251)
0619 06:32:05.666968 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:05.666983 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:05.667020 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:32:05.668674 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:05.668684 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:05.669203 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:05.669487 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:05.670744 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:05.671955 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:05.673295 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:05.674000 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:05.674155 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:05.676528 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:05.676535 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:05.676550 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:05.676584 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:05.679515 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:05.680067 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:05.680073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:05.680078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:05.680228 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:05.680527 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:05.680530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:05.680532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:05.680997 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:05.681283 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:05.681547 (+ 264us) server_negotiation.cc:299] Negotiation successful
0619 06:32:05.681948 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":235,"threads_started":1}
I20260619 06:32:05.683629 17311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:05.766929 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 17188
I20260619 06:32:05.783748 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 17319
I20260619 06:32:05.800396 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:05.784674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44352 (local address 127.0.97.67:39185)
0619 06:32:05.785323 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:05.785337 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:05.785378 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:32:05.787000 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:05.787008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:05.787426 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:05.787708 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:05.788999 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:05.790244 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:05.791457 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:05.792062 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:05.792186 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:05.794854 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:05.794861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:05.794874 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:05.794910 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:05.797717 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:05.798290 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:05.798296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:05.798303 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:05.798444 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:05.798739 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:05.798742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:05.798744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:05.799215 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:05.799426 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:05.799715 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 06:32:05.800093 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":251,"threads_started":1}
I20260619 06:32:05.801789 17442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:05.884503 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 17319
2026-06-19T06:32:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64226 ms)
[----------] 30 tests from SecurityITest (291874 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:05 dist-test-slave-4t1f krb5kdc[17504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:05 dist-test-slave-4t1f krb5kdc[17504](info): set up 2 sockets
Jun 19 06:32:05 dist-test-slave-4t1f krb5kdc[17504](info): commencing operation
krb5kdc: starting...
W20260619 06:32:06.985196 389 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:07 dist-test-slave-4t1f krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:07.210620 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37123
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:37123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:07.530957 17517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:07.531236 17517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:07.531368 17517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:07.538487 17517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:07.538617 17517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:07.538717 17517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:07.538813 17517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:07.538908 17517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:07.548426 17517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:37123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:07.549640 17517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:07.550922 17517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:07.563735 17522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:07.563805 17523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:07.563792 17525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:07.564792 17517 server_base.cc:1034] running on GCE node
I20260619 06:32:07.565562 17517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:07.566845 17517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:07.568064 17517 hybrid_clock.cc:648] HybridClock initialized: now 1781850727568021 us; error 56 us; skew 500 ppm
I20260619 06:32:07.568442 17517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:07 dist-test-slave-4t1f krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850727, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:07.571525 17517 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:07.572996 17517 webserver.cc:460] Webserver started at http://127.0.97.126:37743/ using document root <none> and password file <none>
I20260619 06:32:07.573683 17517 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:07.573827 17517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:07.574127 17517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:07.576892 17517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "df04ae7df2574732a0cd06b2bef7819b"
format_stamp: "Formatted at 2026-06-19 06:32:07 on dist-test-slave-4t1f"
server_key: "7ac58ad08fdb4077ea47da7e98f439a7"
server_key_iv: "0b84ae35cf8e9e746f3cf6a51e17c87d"
server_key_version: "encryptionkey@0"
I20260619 06:32:07.577663 17517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "df04ae7df2574732a0cd06b2bef7819b"
format_stamp: "Formatted at 2026-06-19 06:32:07 on dist-test-slave-4t1f"
server_key: "7ac58ad08fdb4077ea47da7e98f439a7"
server_key_iv: "0b84ae35cf8e9e746f3cf6a51e17c87d"
server_key_version: "encryptionkey@0"
I20260619 06:32:07.582356 17517 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 06:32:07.585665 17532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:07.586813 17517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:07.586992 17517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "df04ae7df2574732a0cd06b2bef7819b"
format_stamp: "Formatted at 2026-06-19 06:32:07 on dist-test-slave-4t1f"
server_key: "7ac58ad08fdb4077ea47da7e98f439a7"
server_key_iv: "0b84ae35cf8e9e746f3cf6a51e17c87d"
server_key_version: "encryptionkey@0"
I20260619 06:32:07.587172 17517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:07.615902 17517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:07.619666 17517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:07.619949 17517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:07.637626 17517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:37123
I20260619 06:32:07.637666 17583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:37123 every 8 connection(s)
I20260619 06:32:07.638942 17517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:07.641610 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 17517
I20260619 06:32:07.641844 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:07.642638 389 external_mini_cluster.cc:1428] Setting key 50efa0faa5f16a5dc06df054b2de138d
I20260619 06:32:07.642923 17584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:07.654642 17584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b: Bootstrap starting.
Jun 19 06:32:07 dist-test-slave-4t1f krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:07.658234 17584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:07.659447 17584 log.cc:824] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:07.662669 17584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b: No bootstrap required, opened a new log
I20260619 06:32:07.665041 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:07.644581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53060 (local address 127.0.97.126:37123)
0619 06:32:07.645373 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:07.645384 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:07.645421 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:32:07.647370 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:07.647390 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:07.647738 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:07.648130 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:07.649651 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:07.650873 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:07.652208 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:07.652893 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:07.653110 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:07.657600 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:07.657607 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:07.657620 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:07.657657 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:07.661104 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:07.661908 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:07.661914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:07.661919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:07.662077 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:07.662475 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:07.662478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:07.662480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:07.662994 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:07.663221 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:07.663702 (+ 481us) server_negotiation.cc:299] Negotiation successful
0619 06:32:07.664042 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":339,"threads_started":1}
I20260619 06:32:07.670516 17584 raft_consensus.cc:348] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df04ae7df2574732a0cd06b2bef7819b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260619 06:32:07.670858 17584 raft_consensus.cc:374] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:07.670974 17584 raft_consensus.cc:729] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df04ae7df2574732a0cd06b2bef7819b, State: Initialized, Role: FOLLOWER
I20260619 06:32:07.671597 17584 consensus_queue.cc:260] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df04ae7df2574732a0cd06b2bef7819b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260619 06:32:07.671833 17584 raft_consensus.cc:388] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:07.671973 17584 raft_consensus.cc:482] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:07.672143 17584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:07.674325 17584 raft_consensus.cc:504] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df04ae7df2574732a0cd06b2bef7819b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260619 06:32:07.674787 17584 leader_election.cc:302] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df04ae7df2574732a0cd06b2bef7819b; no voters:
I20260619 06:32:07.675329 17584 leader_election.cc:288] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:07.675451 17589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:07.676121 17589 raft_consensus.cc:686] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [term 1 LEADER]: Becoming Leader. State: Replica: df04ae7df2574732a0cd06b2bef7819b, State: Running, Role: LEADER
I20260619 06:32:07.676676 17589 consensus_queue.cc:237] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df04ae7df2574732a0cd06b2bef7819b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } }
I20260619 06:32:07.677436 17584 sys_catalog.cc:564] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:07.679652 17591 sys_catalog.cc:455] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [sys.catalog]: SysCatalogTable state changed. Reason: New leader df04ae7df2574732a0cd06b2bef7819b. Latest consensus state: current_term: 1 leader_uuid: "df04ae7df2574732a0cd06b2bef7819b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df04ae7df2574732a0cd06b2bef7819b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } } }
I20260619 06:32:07.679984 17590 sys_catalog.cc:455] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df04ae7df2574732a0cd06b2bef7819b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df04ae7df2574732a0cd06b2bef7819b" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 37123 } } }
I20260619 06:32:07.680222 17591 sys_catalog.cc:458] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:07.680295 17590 sys_catalog.cc:458] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:07.685638 17596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:07.690335 17596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:07.700567 17596 catalog_manager.cc:1347] Generated new cluster ID: 3c119d2d3fb94b80ad399317d44ead89
I20260619 06:32:07.700685 17596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:07.737684 17596 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:07.739018 17596 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:07.743984 17596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b: Generated new TSK 0
I20260619 06:32:07.744736 17596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:32:07.819947 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:08.164935 17610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:08.165282 17610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:08.165452 17610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:08.172566 17610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:08.172695 17610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:08.172879 17610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:32:08.182569 17610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:08.183787 17610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:08.185060 17610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:08.198045 17615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:08.198118 17618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:08.198133 17616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:08.199450 17610 server_base.cc:1034] running on GCE node
I20260619 06:32:08.200078 17610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:08.200968 17610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:08.202159 17610 hybrid_clock.cc:648] HybridClock initialized: now 1781850728202116 us; error 35 us; skew 500 ppm
I20260619 06:32:08.202443 17610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:08 dist-test-slave-4t1f krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:08.205510 17610 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:32:08.206830 17610 webserver.cc:460] Webserver started at http://127.0.97.65:33357/ using document root <none> and password file <none>
I20260619 06:32:08.207482 17610 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:08.207628 17610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:08.207921 17610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:08.210711 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "ec65bc5cea79457ba580acdbc12df9f2"
format_stamp: "Formatted at 2026-06-19 06:32:08 on dist-test-slave-4t1f"
server_key: "f65eeb683946022bbae7030745f50f11"
server_key_iv: "412514746d017b0c2f7ff27f67dac43b"
server_key_version: "encryptionkey@0"
I20260619 06:32:08.211457 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "ec65bc5cea79457ba580acdbc12df9f2"
format_stamp: "Formatted at 2026-06-19 06:32:08 on dist-test-slave-4t1f"
server_key: "f65eeb683946022bbae7030745f50f11"
server_key_iv: "412514746d017b0c2f7ff27f67dac43b"
server_key_version: "encryptionkey@0"
I20260619 06:32:08.216197 17610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:32:08.219398 17625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:08.220614 17610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:32:08.220779 17610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "ec65bc5cea79457ba580acdbc12df9f2"
format_stamp: "Formatted at 2026-06-19 06:32:08 on dist-test-slave-4t1f"
server_key: "f65eeb683946022bbae7030745f50f11"
server_key_iv: "412514746d017b0c2f7ff27f67dac43b"
server_key_version: "encryptionkey@0"
I20260619 06:32:08.220968 17610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:08.261773 17610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:08.265650 17610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:08.265949 17610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:08.266927 17610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:08.268213 17610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:08.268323 17610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:08.268464 17610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:08.268536 17610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:08.301110 17610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:41111
I20260619 06:32:08.301154 17737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:41111 every 8 connection(s)
I20260619 06:32:08.302333 17610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:32:08.307336 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 17610
I20260619 06:32:08.307569 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:32:08.308490 389 external_mini_cluster.cc:1428] Setting key dc74c142136c280190cd292d6fdf253b
Jun 19 06:32:08 dist-test-slave-4t1f krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:08.325250 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:08.304862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:46943 (local address 127.0.97.126:37123)
0619 06:32:08.305360 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:08.305366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:08.306246 (+ 880us) server_negotiation.cc:407] Connection header received
0619 06:32:08.308886 (+ 2640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:08.308891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:08.308956 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:08.309048 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:08.310987 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.311683 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.314404 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.314810 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.314907 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:08.319058 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:08.319062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:08.319065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:08.319094 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:08.322045 (+ 2951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:08.322777 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:08.322785 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:08.322792 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:08.322939 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:08.323346 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:08.323354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:08.323360 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:08.323727 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:08.323886 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:08.324510 (+ 624us) server_negotiation.cc:299] Negotiation successful
0619 06:32:08.324758 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":248,"threads_started":1}
I20260619 06:32:08.326011 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:08.305388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37123 (local address 127.0.97.65:46943)
0619 06:32:08.306081 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:08.306128 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:08.308567 (+ 2439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:08.309354 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:08.309362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:08.310021 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:08.310804 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:08.310830 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.311841 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.311850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:08.314252 (+ 2402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:08.314271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.315037 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.315047 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:08.315385 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:08.316113 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:08.316168 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:08.318829 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:08.322176 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:08.322184 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:08.322199 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:08.322600 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:08.323047 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:08.323050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:08.323053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:08.323213 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:08.323879 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:08.323884 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:08.324147 (+ 263us) client_negotiation.cc:769] Sending connection context
0619 06:32:08.324359 (+ 212us) client_negotiation.cc:240] Negotiation successful
0619 06:32:08.324822 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":268,"threads_started":1}
I20260619 06:32:08.327571 17738 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37123
I20260619 06:32:08.327859 17738 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:08.328534 17738 heartbeater.cc:511] Master 127.0.97.126:37123 requested a full tablet report, sending...
I20260619 06:32:08.330556 17549 ts_manager.cc:194] Registered new tserver with Master: ec65bc5cea79457ba580acdbc12df9f2 (127.0.97.65:41111)
I20260619 06:32:08.331910 17549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:46943
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:32:08.361945 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:08.677393 17744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:08.677680 17744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:08.677784 17744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:08.684674 17744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:08.684830 17744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:08.684998 17744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:32:08.694689 17744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:08.695859 17744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:08.697190 17744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:08.709986 17749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:08.709920 17750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:08.709926 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:08.710825 17744 server_base.cc:1034] running on GCE node
I20260619 06:32:08.711469 17744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:08.712267 17744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:08.713477 17744 hybrid_clock.cc:648] HybridClock initialized: now 1781850728713435 us; error 53 us; skew 500 ppm
I20260619 06:32:08.713775 17744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:08 dist-test-slave-4t1f krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:08.716616 17744 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:32:08.717923 17744 webserver.cc:460] Webserver started at http://127.0.97.66:33309/ using document root <none> and password file <none>
I20260619 06:32:08.718503 17744 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:08.718601 17744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:08.718906 17744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:08.721616 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "82573c4cbcc9451ebaf27986815f6b12"
format_stamp: "Formatted at 2026-06-19 06:32:08 on dist-test-slave-4t1f"
server_key: "33960ecbbfcbd9ed87b9af1612a8a8be"
server_key_iv: "bc3b28501a03462a0ab3e7c5e2b98d47"
server_key_version: "encryptionkey@0"
I20260619 06:32:08.722353 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "82573c4cbcc9451ebaf27986815f6b12"
format_stamp: "Formatted at 2026-06-19 06:32:08 on dist-test-slave-4t1f"
server_key: "33960ecbbfcbd9ed87b9af1612a8a8be"
server_key_iv: "bc3b28501a03462a0ab3e7c5e2b98d47"
server_key_version: "encryptionkey@0"
I20260619 06:32:08.727396 17744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:32:08.730592 17759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:08.731698 17744 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:32:08.731844 17744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "82573c4cbcc9451ebaf27986815f6b12"
format_stamp: "Formatted at 2026-06-19 06:32:08 on dist-test-slave-4t1f"
server_key: "33960ecbbfcbd9ed87b9af1612a8a8be"
server_key_iv: "bc3b28501a03462a0ab3e7c5e2b98d47"
server_key_version: "encryptionkey@0"
I20260619 06:32:08.732018 17744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:08.756915 17744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:08.760761 17744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:08.761041 17744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:08.761998 17744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:08.763219 17744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:08.763294 17744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:08.763454 17744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:08.763501 17744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:08.795800 17744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:37405
I20260619 06:32:08.795846 17871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:37405 every 8 connection(s)
I20260619 06:32:08.796990 17744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:32:08.802835 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 17744
I20260619 06:32:08.803025 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:32:08.803450 389 external_mini_cluster.cc:1428] Setting key 19bc24e195e1f3c7ad93853c38828294
Jun 19 06:32:08 dist-test-slave-4t1f krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850728, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:08.818598 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:08.799470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:42389 (local address 127.0.97.126:37123)
0619 06:32:08.799671 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:08.799678 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:08.800802 (+ 1124us) server_negotiation.cc:407] Connection header received
0619 06:32:08.802621 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:08.802629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:08.802719 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:08.802839 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:08.804597 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.805623 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.808004 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.808398 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.808485 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:08.812481 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:08.812490 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:08.812498 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:08.812541 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:08.815425 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:08.816374 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:08.816378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:08.816380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:08.816490 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:08.816985 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:08.816987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:08.816989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:08.817307 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:08.817423 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:08.818202 (+ 779us) server_negotiation.cc:299] Negotiation successful
0619 06:32:08.818365 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:32:08.820204 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:08.799995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37123 (local address 127.0.97.66:42389)
0619 06:32:08.800644 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:08.800680 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:08.802352 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:08.803106 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:08.803118 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:08.803676 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:08.804434 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:08.804464 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.805773 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.805778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:08.807855 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:08.807877 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:08.808599 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:08.808623 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:08.808944 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:08.809656 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:08.809688 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:08.812253 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:08.815650 (+ 3397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:08.815661 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:08.815677 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:08.816270 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:08.816587 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:08.816595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:08.816601 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:08.816861 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:08.817425 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:08.817436 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:08.817831 (+ 395us) client_negotiation.cc:769] Sending connection context
0619 06:32:08.818139 (+ 308us) client_negotiation.cc:240] Negotiation successful
0619 06:32:08.818682 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":251,"threads_started":1}
I20260619 06:32:08.821619 17872 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37123
I20260619 06:32:08.821907 17872 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:08.822592 17872 heartbeater.cc:511] Master 127.0.97.126:37123 requested a full tablet report, sending...
I20260619 06:32:08.824126 17549 ts_manager.cc:194] Registered new tserver with Master: 82573c4cbcc9451ebaf27986815f6b12 (127.0.97.66:37405)
I20260619 06:32:08.825361 17549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:42389
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:32:08.857867 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:45265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:09.172515 17877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:09.172842 17877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:09.173008 17877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:09.180406 17877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:09.180549 17877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:09.180713 17877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:32:09.191039 17877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:45265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:37123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:09.192348 17877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:09.193674 17877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:09.206194 17882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:09.206245 17883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:09.206658 17885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:09.210363 17877 server_base.cc:1034] running on GCE node
I20260619 06:32:09.210944 17877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:09.211735 17877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:09.212951 17877 hybrid_clock.cc:648] HybridClock initialized: now 1781850729212906 us; error 58 us; skew 500 ppm
I20260619 06:32:09.213280 17877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:09 dist-test-slave-4t1f krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:09.216284 17877 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:32:09.217713 17877 webserver.cc:460] Webserver started at http://127.0.97.67:35585/ using document root <none> and password file <none>
I20260619 06:32:09.218384 17877 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:09.218523 17877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:09.218820 17877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:09.221704 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c"
format_stamp: "Formatted at 2026-06-19 06:32:09 on dist-test-slave-4t1f"
server_key: "e7df621c440566f0ae05ec265304fecd"
server_key_iv: "c9beda2bf4c9fedff3e984e487e3d5c8"
server_key_version: "encryptionkey@0"
I20260619 06:32:09.222443 17877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c"
format_stamp: "Formatted at 2026-06-19 06:32:09 on dist-test-slave-4t1f"
server_key: "e7df621c440566f0ae05ec265304fecd"
server_key_iv: "c9beda2bf4c9fedff3e984e487e3d5c8"
server_key_version: "encryptionkey@0"
I20260619 06:32:09.227411 17877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:32:09.230608 17892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:09.231793 17877 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:32:09.231961 17877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c"
format_stamp: "Formatted at 2026-06-19 06:32:09 on dist-test-slave-4t1f"
server_key: "e7df621c440566f0ae05ec265304fecd"
server_key_iv: "c9beda2bf4c9fedff3e984e487e3d5c8"
server_key_version: "encryptionkey@0"
I20260619 06:32:09.232141 17877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:09.251494 17877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:09.255239 17877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:09.255519 17877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:09.256381 17877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:09.257737 17877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:09.257846 17877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:09.257990 17877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:09.258062 17877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:09.290961 17877 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:37523
I20260619 06:32:09.291020 18004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:37523 every 8 connection(s)
I20260619 06:32:09.292297 17877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:32:09.299333 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 17877
I20260619 06:32:09.299530 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:32:09.299969 389 external_mini_cluster.cc:1428] Setting key cdf548366e2f4cda842fc60c792ed4e7
Jun 19 06:32:09 dist-test-slave-4t1f krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850729, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:09.312623 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.294815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:38939 (local address 127.0.97.126:37123)
0619 06:32:09.295004 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.295009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.296197 (+ 1188us) server_negotiation.cc:407] Connection header received
0619 06:32:09.298140 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.298146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.298269 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.298383 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:09.300020 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.301118 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.303503 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.304145 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.304276 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.307484 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:09.307489 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:09.307492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:09.307522 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:09.310055 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:09.310706 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:09.310710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:09.310713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:09.310852 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:09.311250 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:09.311253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:09.311256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:09.311511 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:09.311637 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.312216 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.312371 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:32:09.313550 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.295315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:37123 (local address 127.0.97.67:38939)
0619 06:32:09.296031 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:09.296068 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:09.297867 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:09.298649 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:09.298661 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:09.299201 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:09.299851 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.299868 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.301637 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.301643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.303366 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.303378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.304372 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.304380 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.304577 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.305248 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:09.305275 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:09.307306 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:09.310199 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:09.310205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:09.310215 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:09.310586 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:09.310949 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:09.310968 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:09.310971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:09.311147 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:09.311626 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:09.311631 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:09.311902 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 06:32:09.312110 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 06:32:09.312436 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":265,"threads_started":1}
I20260619 06:32:09.314924 18005 heartbeater.cc:347] Connected to a master server at 127.0.97.126:37123
I20260619 06:32:09.315253 18005 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:09.315917 18005 heartbeater.cc:511] Master 127.0.97.126:37123 requested a full tablet report, sending...
I20260619 06:32:09.317317 17549 ts_manager.cc:194] Registered new tserver with Master: d2d2986d0f4143f7ba41d8ff7aef5f2c (127.0.97.67:37523)
I20260619 06:32:09.318033 17549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:38939
I20260619 06:32:09.327361 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:32:09.334666 17738 heartbeater.cc:503] Master 127.0.97.126:37123 was elected leader, sending a full tablet report...
I20260619 06:32:09.344895 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.331636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.0.97.126:37123)
0619 06:32:09.331811 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.331819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.332194 (+ 375us) server_negotiation.cc:407] Connection header received
0619 06:32:09.332378 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.332383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.332482 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.332580 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:09.334125 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.334978 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.336680 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.337210 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.337343 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.339655 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:09.339661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:09.339667 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:09.339701 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:09.342324 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:09.342979 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:09.342986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:09.342989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:09.343118 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:09.343489 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:09.343494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:09.343498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:09.343793 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:09.344063 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.344376 (+ 313us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.344580 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 06:32:09.348897 17549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:32:09.355919 17549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:32:09.378080 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.368068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49156 (local address 127.0.97.66:37405)
0619 06:32:09.369321 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.369330 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.369359 (+ 29us) server_negotiation.cc:407] Connection header received
0619 06:32:09.369459 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.369466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.369711 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.370693 (+ 982us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:09.371305 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.372881 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.375747 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.376744 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.376900 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.377121 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.377577 (+ 456us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.377746 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":987,"threads_started":1}
I20260619 06:32:09.378175 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.367934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37405 (local address 127.0.0.1:49156)
0619 06:32:09.368813 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:09.368868 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:09.369103 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:09.369943 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:09.369951 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:09.369980 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:09.370442 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.370461 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.373040 (+ 2579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.373047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.375584 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.375594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.376950 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.376956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.377348 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.377402 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 06:32:09.377535 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 06:32:09.377714 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":101,"thread_start_us":347,"threads_started":1}
I20260619 06:32:09.378885 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.369531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.97.65:41111)
0619 06:32:09.370083 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.370092 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.370132 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:32:09.370235 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.370244 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.370474 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.370681 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:09.372392 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.373768 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.376019 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.377494 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.377614 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.377829 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.378368 (+ 539us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.378525 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":550,"threads_started":1}
I20260619 06:32:09.380570 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.370336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.0.97.67:37523)
0619 06:32:09.372386 (+ 2050us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.372394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.372422 (+ 28us) server_negotiation.cc:407] Connection header received
0619 06:32:09.372503 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.372511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.372699 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.372884 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:09.373563 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.375422 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.378027 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.379609 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.379748 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.379972 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.380069 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.380227 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1884,"thread_start_us":1743,"threads_started":1}
I20260619 06:32:09.379530 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.369055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41111 (local address 127.0.0.1:40276)
0619 06:32:09.369604 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:09.369633 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:09.369748 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:09.371723 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:09.371730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:09.371758 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:09.372163 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.372173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.373885 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.373890 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.375849 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.375863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.377971 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.377978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.378142 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.378181 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 06:32:09.379068 (+ 887us) client_negotiation.cc:240] Negotiation successful
0619 06:32:09.379205 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":580,"threads_started":1}
I20260619 06:32:09.381624 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.369959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37523 (local address 127.0.0.1:60182)
0619 06:32:09.371415 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:09.371445 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:09.371568 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:09.372900 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:09.372905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:09.372931 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:09.373379 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.373390 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.375589 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.375593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.377834 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.377853 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.379832 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.379837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.379944 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.379965 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:32:09.380035 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:32:09.380109 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1276,"mutex_wait_us":30,"thread_start_us":1111,"threads_started":1}
I20260619 06:32:09.382387 17807 tablet_service.cc:1467] Processing CreateTablet for tablet 0adf71744b574467ab4d8410692aaad1 (DEFAULT_TABLE table=test-table [id=c3f7979ca18d42bf87b6a0443a62dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:09.383816 17807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0adf71744b574467ab4d8410692aaad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:09.384512 17673 tablet_service.cc:1467] Processing CreateTablet for tablet 0adf71744b574467ab4d8410692aaad1 (DEFAULT_TABLE table=test-table [id=c3f7979ca18d42bf87b6a0443a62dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:09.385042 17940 tablet_service.cc:1467] Processing CreateTablet for tablet 0adf71744b574467ab4d8410692aaad1 (DEFAULT_TABLE table=test-table [id=c3f7979ca18d42bf87b6a0443a62dea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:09.385601 17673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0adf71744b574467ab4d8410692aaad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:09.386324 17940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0adf71744b574467ab4d8410692aaad1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:09.397545 18023 tablet_bootstrap.cc:492] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Bootstrap starting.
I20260619 06:32:09.400286 18023 tablet_bootstrap.cc:654] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:09.401561 18023 log.cc:824] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:09.402429 18025 tablet_bootstrap.cc:492] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Bootstrap starting.
I20260619 06:32:09.401662 18024 tablet_bootstrap.cc:492] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Bootstrap starting.
I20260619 06:32:09.405001 18023 tablet_bootstrap.cc:492] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: No bootstrap required, opened a new log
I20260619 06:32:09.405292 18023 ts_tablet_manager.cc:1397] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:32:09.405906 18025 tablet_bootstrap.cc:654] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:09.406165 18024 tablet_bootstrap.cc:654] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:09.406881 18025 log.cc:824] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:09.407312 18024 log.cc:824] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:09.409775 18024 tablet_bootstrap.cc:492] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: No bootstrap required, opened a new log
I20260619 06:32:09.409830 18025 tablet_bootstrap.cc:492] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: No bootstrap required, opened a new log
I20260619 06:32:09.410032 18024 ts_tablet_manager.cc:1397] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 06:32:09.410075 18025 ts_tablet_manager.cc:1397] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260619 06:32:09.415954 18023 raft_consensus.cc:348] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.416389 18023 raft_consensus.cc:374] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:09.416487 18023 raft_consensus.cc:729] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec65bc5cea79457ba580acdbc12df9f2, State: Initialized, Role: FOLLOWER
I20260619 06:32:09.417104 18023 consensus_queue.cc:260] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.417203 18024 raft_consensus.cc:348] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.417564 18024 raft_consensus.cc:374] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:09.417650 18024 raft_consensus.cc:729] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82573c4cbcc9451ebaf27986815f6b12, State: Initialized, Role: FOLLOWER
I20260619 06:32:09.418233 18024 consensus_queue.cc:260] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.419010 18023 ts_tablet_manager.cc:1428] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260619 06:32:09.419842 17872 heartbeater.cc:503] Master 127.0.97.126:37123 was elected leader, sending a full tablet report...
I20260619 06:32:09.420504 18024 ts_tablet_manager.cc:1428] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Time spent starting tablet: real 0.010s user 0.008s sys 0.002s
I20260619 06:32:09.420331 18025 raft_consensus.cc:348] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.420698 18025 raft_consensus.cc:374] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:09.420785 18025 raft_consensus.cc:729] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d2986d0f4143f7ba41d8ff7aef5f2c, State: Initialized, Role: FOLLOWER
I20260619 06:32:09.421379 18025 consensus_queue.cc:260] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.422410 18005 heartbeater.cc:503] Master 127.0.97.126:37123 was elected leader, sending a full tablet report...
I20260619 06:32:09.422966 18025 ts_tablet_manager.cc:1428] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
W20260619 06:32:09.543967 18006 tablet.cc:2369] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:32:09.548980 17873 tablet.cc:2369] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:32:09.554733 17739 tablet.cc:2369] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:09.798457 18031 raft_consensus.cc:482] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:32:09.798758 18031 raft_consensus.cc:504] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.800379 18031 leader_election.cc:288] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82573c4cbcc9451ebaf27986815f6b12 (127.0.97.66:37405), ec65bc5cea79457ba580acdbc12df9f2 (127.0.97.65:41111)
I20260619 06:32:09.807384 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.801931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:40051 (local address 127.0.97.65:41111)
0619 06:32:09.802139 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.802145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.802162 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:32:09.802267 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.802274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.802350 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.802460 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:09.802970 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.803702 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.805448 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.806408 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.806542 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.806639 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.806998 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.807131 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:32:09.807459 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.801123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:41111 (local address 127.0.97.67:40051)
0619 06:32:09.801995 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:09.802011 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:09.802146 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:09.802473 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:09.802477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:09.802498 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:09.802809 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.802816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.803838 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.803843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.805306 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.805317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.806620 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.806628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.806820 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.806858 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 06:32:09.806961 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:32:09.807094 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":611,"threads_started":1}
I20260619 06:32:09.808414 18007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.800716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:37405 (local address 127.0.97.67:52539)
0619 06:32:09.800912 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:09.800933 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:09.801095 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:09.801954 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:09.801962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:09.802003 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:09.802298 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.802311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.803958 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.803973 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.805992 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:09.806007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.807704 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.807710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:09.807906 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.807940 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:32:09.808031 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:32:09.808160 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"spinlock_wait_cycles":29568}
I20260619 06:32:09.808480 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.800895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:52539 (local address 127.0.97.66:37405)
0619 06:32:09.801105 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.801110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.801132 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:32:09.801201 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.801207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.801276 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.801362 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:09.802513 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.803775 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.806167 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.807268 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.807409 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.807493 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.808085 (+ 592us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.808218 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:32:09.808542 17693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0adf71744b574467ab4d8410692aaad1" candidate_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65bc5cea79457ba580acdbc12df9f2" is_pre_election: true
I20260619 06:32:09.809154 17693 raft_consensus.cc:2454] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d2986d0f4143f7ba41d8ff7aef5f2c in term 0.
I20260619 06:32:09.809844 17896 leader_election.cc:302] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2d2986d0f4143f7ba41d8ff7aef5f2c, ec65bc5cea79457ba580acdbc12df9f2; no voters:
I20260619 06:32:09.809736 17827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0adf71744b574467ab4d8410692aaad1" candidate_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82573c4cbcc9451ebaf27986815f6b12" is_pre_election: true
I20260619 06:32:09.810204 18031 raft_consensus.cc:2781] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:32:09.810235 17827 raft_consensus.cc:2454] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2d2986d0f4143f7ba41d8ff7aef5f2c in term 0.
I20260619 06:32:09.810353 18031 raft_consensus.cc:482] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:32:09.810483 18031 raft_consensus.cc:3037] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:09.812626 18031 raft_consensus.cc:504] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.813222 18031 leader_election.cc:288] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [CANDIDATE]: Term 1 election: Requested vote from peers 82573c4cbcc9451ebaf27986815f6b12 (127.0.97.66:37405), ec65bc5cea79457ba580acdbc12df9f2 (127.0.97.65:41111)
I20260619 06:32:09.813622 17827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0adf71744b574467ab4d8410692aaad1" candidate_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82573c4cbcc9451ebaf27986815f6b12"
I20260619 06:32:09.813700 17693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0adf71744b574467ab4d8410692aaad1" candidate_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec65bc5cea79457ba580acdbc12df9f2"
I20260619 06:32:09.813817 17827 raft_consensus.cc:3037] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:09.813894 17693 raft_consensus.cc:3037] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:09.815943 17693 raft_consensus.cc:2454] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d2986d0f4143f7ba41d8ff7aef5f2c in term 1.
I20260619 06:32:09.816200 17827 raft_consensus.cc:2454] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2d2986d0f4143f7ba41d8ff7aef5f2c in term 1.
I20260619 06:32:09.816299 17896 leader_election.cc:302] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2d2986d0f4143f7ba41d8ff7aef5f2c, ec65bc5cea79457ba580acdbc12df9f2; no voters:
I20260619 06:32:09.816687 18031 raft_consensus.cc:2781] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:09.817198 18031 raft_consensus.cc:686] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 1 LEADER]: Becoming Leader. State: Replica: d2d2986d0f4143f7ba41d8ff7aef5f2c, State: Running, Role: LEADER
I20260619 06:32:09.817706 18031 consensus_queue.cc:237] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } }
I20260619 06:32:09.822341 17548 catalog_manager.cc:5495] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c reported cstate change: term changed from 0 to 1, leader changed from <none> to d2d2986d0f4143f7ba41d8ff7aef5f2c (127.0.97.67). New cstate: current_term: 1 leader_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2d2986d0f4143f7ba41d8ff7aef5f2c" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37523 } health_report { overall_health: HEALTHY } } }
I20260619 06:32:09.939927 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.934049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.0.97.67:37523)
0619 06:32:09.934537 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.934542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.934631 (+ 89us) server_negotiation.cc:407] Connection header received
0619 06:32:09.934850 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.934854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.934953 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.935077 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:32:09.935523 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.936333 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.937804 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.938225 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.938364 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:09.938814 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:32:09.939309 (+ 495us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:32:09.939460 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:09.939556 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 06:32:09.939643 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":273,"threads_started":1}
I20260619 06:32:09.947589 17693 raft_consensus.cc:1261] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 1 FOLLOWER]: Refusing update from remote peer d2d2986d0f4143f7ba41d8ff7aef5f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:09.947590 17827 raft_consensus.cc:1261] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 1 FOLLOWER]: Refusing update from remote peer d2d2986d0f4143f7ba41d8ff7aef5f2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:09.948514 18033 consensus_queue.cc:1035] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec65bc5cea79457ba580acdbc12df9f2" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 41111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:09.948879 18031 consensus_queue.cc:1035] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "82573c4cbcc9451ebaf27986815f6b12" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 37405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:09.972896 17547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:09.973189 17547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:09.976378 17547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df04ae7df2574732a0cd06b2bef7819b: Sending DeleteTablet for 3 replicas of tablet 0adf71744b574467ab4d8410692aaad1
I20260619 06:32:09.977511 17807 tablet_service.cc:1514] Processing DeleteTablet for tablet 0adf71744b574467ab4d8410692aaad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:09 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49156
I20260619 06:32:09.978032 17673 tablet_service.cc:1514] Processing DeleteTablet for tablet 0adf71744b574467ab4d8410692aaad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:09 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:40276
I20260619 06:32:09.978091 17940 tablet_service.cc:1514] Processing DeleteTablet for tablet 0adf71744b574467ab4d8410692aaad1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:09 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:60182
I20260619 06:32:09.979071 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 17610
I20260619 06:32:09.988444 18052 tablet_replica.cc:330] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: stopping tablet replica
I20260619 06:32:09.988659 18051 tablet_replica.cc:330] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: stopping tablet replica
I20260619 06:32:09.989192 18051 raft_consensus.cc:2229] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:09.989742 18051 raft_consensus.cc:2258] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:09.989701 18052 raft_consensus.cc:2229] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:32:09.990952 18052 raft_consensus.cc:2258] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:09.992015 18051 ts_tablet_manager.cc:1905] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:09.993604 18052 ts_tablet_manager.cc:1905] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:09.994642 18050 tablet_replica.cc:330] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: stopping tablet replica
I20260619 06:32:09.995133 18050 raft_consensus.cc:2229] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:09.995585 18050 raft_consensus.cc:2258] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:09.997357 18050 ts_tablet_manager.cc:1905] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:09.999342 18051 ts_tablet_manager.cc:1918] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:09.999512 18051 log.cc:1196] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/0adf71744b574467ab4d8410692aaad1
I20260619 06:32:10.000010 18051 ts_tablet_manager.cc:1939] T 0adf71744b574467ab4d8410692aaad1 P ec65bc5cea79457ba580acdbc12df9f2: Deleting consensus metadata
Jun 19 06:32:09 dist-test-slave-4t1f krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:32:10.001982 17536 catalog_manager.cc:4841] TS ec65bc5cea79457ba580acdbc12df9f2 (127.0.97.65:41111): tablet 0adf71744b574467ab4d8410692aaad1 (table test-table [id=c3f7979ca18d42bf87b6a0443a62dea1]) successfully deleted
I20260619 06:32:10.005579 18052 ts_tablet_manager.cc:1918] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:10.005771 18052 log.cc:1196] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/0adf71744b574467ab4d8410692aaad1
I20260619 06:32:10.006340 18052 ts_tablet_manager.cc:1939] T 0adf71744b574467ab4d8410692aaad1 P d2d2986d0f4143f7ba41d8ff7aef5f2c: Deleting consensus metadata
I20260619 06:32:10.006812 18050 ts_tablet_manager.cc:1918] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:10.007023 18050 log.cc:1196] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/0adf71744b574467ab4d8410692aaad1
I20260619 06:32:10.007587 18050 ts_tablet_manager.cc:1939] T 0adf71744b574467ab4d8410692aaad1 P 82573c4cbcc9451ebaf27986815f6b12: Deleting consensus metadata
I20260619 06:32:10.008108 17536 catalog_manager.cc:4841] TS d2d2986d0f4143f7ba41d8ff7aef5f2c (127.0.97.67:37523): tablet 0adf71744b574467ab4d8410692aaad1 (table test-table [id=c3f7979ca18d42bf87b6a0443a62dea1]) successfully deleted
I20260619 06:32:10.009370 17536 catalog_manager.cc:4841] TS 82573c4cbcc9451ebaf27986815f6b12 (127.0.97.66:37405): tablet 0adf71744b574467ab4d8410692aaad1 (table test-table [id=c3f7979ca18d42bf87b6a0443a62dea1]) successfully deleted
I20260619 06:32:10.011226 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:09.980847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.97.65:41111)
0619 06:32:09.981091 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:09.981098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:09.986396 (+ 5298us) server_negotiation.cc:407] Connection header received
0619 06:32:09.986607 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:09.986616 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:09.986720 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:09.986860 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:09.990230 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.991406 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.993365 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:09.994054 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:09.994148 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:10.003305 (+ 9157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:10.003320 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:10.003342 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:10.003393 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:10.007337 (+ 3944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:10.008832 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:10.008843 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:10.008846 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:10.009014 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:10.009615 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:10.009620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:10.009622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:10.009987 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:10.010317 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:10.010626 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 06:32:10.010856 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:32:10.011991 17733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:10.162518 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 17610
I20260619 06:32:10.185834 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 17744
Jun 19 06:32:10 dist-test-slave-4t1f krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:32:10.202580 18018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:10.186602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49164 (local address 127.0.97.66:37405)
0619 06:32:10.186808 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:10.186814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:10.186838 (+ 24us) server_negotiation.cc:407] Connection header received
0619 06:32:10.186934 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:10.186952 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:10.187043 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:10.187180 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:10.188352 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:10.189503 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:10.190779 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:10.191477 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:10.191613 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:10.196068 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:10.196086 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:10.196105 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:10.196143 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:10.199931 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:10.200552 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:10.200562 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:10.200566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:10.200698 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:10.201036 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:10.201041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:10.201044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:10.201537 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:10.201862 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:10.202008 (+ 146us) server_negotiation.cc:299] Negotiation successful
0619 06:32:10.202215 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:32:10.203325 17867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:10.291057 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 17744
I20260619 06:32:10.305871 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 17877
Jun 19 06:32:10 dist-test-slave-4t1f krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850727, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:32:10.321821 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:10.306625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.0.97.67:37523)
0619 06:32:10.306808 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:10.306814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:10.306964 (+ 150us) server_negotiation.cc:407] Connection header received
0619 06:32:10.307178 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:10.307181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:10.307258 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:10.307375 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:10.308758 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:10.309736 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:10.310992 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:10.311438 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:10.311595 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:10.315858 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:10.315867 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:10.315880 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:10.315909 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:10.318958 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:10.319615 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:10.319632 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:10.319635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:10.319758 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:10.320137 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:10.320147 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:10.320149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:10.320491 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:10.320847 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:10.321051 (+ 204us) server_negotiation.cc:299] Negotiation successful
0619 06:32:10.321464 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:32:10.322495 18000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:10.441565 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 17877
I20260619 06:32:10.460156 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 17517
I20260619 06:32:10.461315 17579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:10.571538 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 17517
2026-06-19T06:32:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4692 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T06:32:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:10Z Disabled control of system clock
I20260619 06:32:10.646332 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46317
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:46317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:32:10.962023 18061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:10.962286 18061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:10.962416 18061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:10.969481 18061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:10.969574 18061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:10.969658 18061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:10.969764 18061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:10.969842 18061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:10.978689 18061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:46317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:46317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:10.979861 18061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:10.981686 18061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:10.993144 18069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:10.993223 18066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:10.993244 18067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:10.993929 18061 server_base.cc:1034] running on GCE node
I20260619 06:32:10.994658 18061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:10.995903 18061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:10.997157 18061 hybrid_clock.cc:648] HybridClock initialized: now 1781850730997090 us; error 48 us; skew 500 ppm
I20260619 06:32:10.997483 18061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:11.004648 18061 webserver.cc:460] Webserver started at http://127.0.97.126:33967/ using document root <none> and password file <none>
I20260619 06:32:11.005364 18061 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:11.005492 18061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:11.005807 18061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:11.008512 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "b7547017c9a340cc8f8166943602b46c"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "2ed46d7802685b47a0af79782f982be9"
server_key_iv: "84d323f4fa9ed5ad910b2b133e9ef7c3"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.009336 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "b7547017c9a340cc8f8166943602b46c"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "2ed46d7802685b47a0af79782f982be9"
server_key_iv: "84d323f4fa9ed5ad910b2b133e9ef7c3"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.013943 18061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:32:11.016983 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:11.018281 18061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:32:11.018457 18061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "b7547017c9a340cc8f8166943602b46c"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "2ed46d7802685b47a0af79782f982be9"
server_key_iv: "84d323f4fa9ed5ad910b2b133e9ef7c3"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.018652 18061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:11.050580 18061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:11.051391 18061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:11.051680 18061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:11.069476 18061 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:46317
I20260619 06:32:11.069499 18126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:46317 every 8 connection(s)
I20260619 06:32:11.070760 18061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:11.074342 18127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:11.075259 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18061
I20260619 06:32:11.075541 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:11.075989 389 external_mini_cluster.cc:1428] Setting key 04fe47522842716d8a85535205b201c3
I20260619 06:32:11.088172 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c: Bootstrap starting.
I20260619 06:32:11.090709 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:11.077857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34902 (local address 127.0.97.126:46317)
0619 06:32:11.078902 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:11.078916 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:11.078965 (+ 49us) server_negotiation.cc:407] Connection header received
0619 06:32:11.080866 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:11.080890 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:11.081533 (+ 643us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:11.081961 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:11.082455 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:11.084241 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:11.085833 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:11.086828 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:11.086996 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:11.088024 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:11.088035 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:11.088051 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:11.088740 (+ 689us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:11.088837 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:11.088862 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:11.088979 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:11.089177 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 06:32:11.089467 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260619 06:32:11.092031 18127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:11.093030 18127 log.cc:824] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:11.095624 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c: No bootstrap required, opened a new log
I20260619 06:32:11.102741 18127 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7547017c9a340cc8f8166943602b46c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46317 } }
I20260619 06:32:11.103068 18127 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:11.103188 18127 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7547017c9a340cc8f8166943602b46c, State: Initialized, Role: FOLLOWER
I20260619 06:32:11.103706 18127 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7547017c9a340cc8f8166943602b46c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46317 } }
I20260619 06:32:11.103909 18127 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:11.104061 18127 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:11.104225 18127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:11.106388 18127 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7547017c9a340cc8f8166943602b46c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46317 } }
I20260619 06:32:11.106926 18127 leader_election.cc:302] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7547017c9a340cc8f8166943602b46c; no voters:
I20260619 06:32:11.107486 18127 leader_election.cc:288] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:11.107615 18132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:11.108405 18132 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [term 1 LEADER]: Becoming Leader. State: Replica: b7547017c9a340cc8f8166943602b46c, State: Running, Role: LEADER
I20260619 06:32:11.108879 18132 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7547017c9a340cc8f8166943602b46c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46317 } }
I20260619 06:32:11.109544 18127 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:11.111824 18134 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7547017c9a340cc8f8166943602b46c. Latest consensus state: current_term: 1 leader_uuid: "b7547017c9a340cc8f8166943602b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7547017c9a340cc8f8166943602b46c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46317 } } }
I20260619 06:32:11.112164 18134 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:11.113113 18140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:11.112902 18133 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7547017c9a340cc8f8166943602b46c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7547017c9a340cc8f8166943602b46c" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 46317 } } }
I20260619 06:32:11.113313 18133 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:11.119673 18140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:11.128304 18140 catalog_manager.cc:1347] Generated new cluster ID: b5421002688a425db97f5bfff3d0ebd5
I20260619 06:32:11.128404 18140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:11.150538 18140 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:11.152549 18140 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:11.176592 18140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c: Generated new TSK 0
I20260619 06:32:11.177742 18140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:11.186193 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46317
--builtin_ntp_servers=127.0.97.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:32:11.509960 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:11.510237 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:11.510403 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:11.517745 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:11.517854 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:11.518028 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:32:11.527130 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:11.528383 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:11.530211 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:11.542836 18157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:11.542794 18156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:11.542766 18159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:11.543787 18151 server_base.cc:1034] running on GCE node
I20260619 06:32:11.544430 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:11.545365 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:11.546557 18151 hybrid_clock.cc:648] HybridClock initialized: now 1781850731546511 us; error 37 us; skew 500 ppm
I20260619 06:32:11.546866 18151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:11.549026 18151 webserver.cc:460] Webserver started at http://127.0.97.65:42047/ using document root <none> and password file <none>
I20260619 06:32:11.549734 18151 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:11.549849 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:11.550168 18151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:11.552898 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "5ce80a9a347e6210e5f2772e7d606578"
server_key_iv: "ca2d70b62185d0ed907fbc29cb51f2f0"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.553719 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "5ce80a9a347e6210e5f2772e7d606578"
server_key_iv: "ca2d70b62185d0ed907fbc29cb51f2f0"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.558370 18151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 06:32:11.561648 18165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:11.562955 18151 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:32:11.563138 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "5ce80a9a347e6210e5f2772e7d606578"
server_key_iv: "ca2d70b62185d0ed907fbc29cb51f2f0"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.563354 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:11.589869 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:11.590734 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:11.590955 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:11.591933 18151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:11.593336 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:11.593422 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:11.593497 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:11.593554 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:11.625459 18151 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36889
I20260619 06:32:11.625500 18277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36889 every 8 connection(s)
I20260619 06:32:11.626685 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:32:11.632092 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18151
I20260619 06:32:11.632267 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:32:11.632686 389 external_mini_cluster.cc:1428] Setting key 76c220b01e54483acfd85d04574a4f52
I20260619 06:32:11.635138 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46317
--builtin_ntp_servers=127.0.97.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:32:11.638675 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:11.628963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:54527 (local address 127.0.97.126:46317)
0619 06:32:11.629446 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:11.629450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:11.630463 (+ 1013us) server_negotiation.cc:407] Connection header received
0619 06:32:11.632256 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:11.632261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:11.632364 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:11.632492 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:11.633617 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:11.634351 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:11.635916 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:11.636304 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:11.636429 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:11.637846 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:11.637859 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:11.637863 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:11.637879 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:11.637918 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:11.637929 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:11.638024 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:11.638252 (+ 228us) server_negotiation.cc:299] Negotiation successful
0619 06:32:11.638344 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":290,"threads_started":1}
I20260619 06:32:11.639621 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:11.629457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46317 (local address 127.0.97.65:54527)
0619 06:32:11.630316 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:11.630350 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:11.631993 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:11.632731 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:11.632739 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:11.632868 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:11.633455 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:11.633471 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:11.634488 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:11.634500 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:11.635787 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:11.635799 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:11.636522 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:11.636530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:11.636782 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:11.637475 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:11.637477 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:11.637514 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:11.637588 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:11.638023 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:11.638036 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:11.638045 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:32:11.638217 (+ 172us) client_negotiation.cc:240] Negotiation successful
0619 06:32:11.638585 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":269,"threads_started":1}
I20260619 06:32:11.641113 18278 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46317
I20260619 06:32:11.641407 18278 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:11.642236 18278 heartbeater.cc:511] Master 127.0.97.126:46317 requested a full tablet report, sending...
I20260619 06:32:11.644129 18092 ts_manager.cc:194] Registered new tserver with Master: 4c875dd2c1d7474b992cf76a8d0c6bb1 (127.0.97.65:36889)
I20260619 06:32:11.645587 18092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:54527
W20260619 06:32:11.951886 18282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:11.952174 18282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:11.952342 18282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:11.959350 18282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:11.959473 18282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:11.959649 18282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:32:11.968566 18282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:11.969758 18282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:11.971437 18282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:11.983939 18287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:11.983851 18290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:11.983906 18288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:11.984752 18282 server_base.cc:1034] running on GCE node
I20260619 06:32:11.985419 18282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:11.986315 18282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:11.987536 18282 hybrid_clock.cc:648] HybridClock initialized: now 1781850731987502 us; error 60 us; skew 500 ppm
I20260619 06:32:11.987840 18282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:11.990173 18282 webserver.cc:460] Webserver started at http://127.0.97.66:38715/ using document root <none> and password file <none>
I20260619 06:32:11.990821 18282 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:11.990947 18282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:11.991271 18282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:11.993975 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "6fd326725dd8ddb298c1a57924864be5"
server_key_iv: "c32fb0031afc06ab63453f0eb36dd2f4"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.994725 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "6fd326725dd8ddb298c1a57924864be5"
server_key_iv: "c32fb0031afc06ab63453f0eb36dd2f4"
server_key_version: "encryptionkey@0"
I20260619 06:32:11.999439 18282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:32:12.002609 18296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:12.003754 18282 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:32:12.003917 18282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd"
format_stamp: "Formatted at 2026-06-19 06:32:11 on dist-test-slave-4t1f"
server_key: "6fd326725dd8ddb298c1a57924864be5"
server_key_iv: "c32fb0031afc06ab63453f0eb36dd2f4"
server_key_version: "encryptionkey@0"
I20260619 06:32:12.004122 18282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:12.017572 18282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:12.018253 18282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:12.018535 18282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:12.019413 18282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:12.020718 18282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:12.020841 18282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:12.020958 18282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:12.021041 18282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:12.052563 18282 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43143
I20260619 06:32:12.052613 18408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43143 every 8 connection(s)
I20260619 06:32:12.053781 18282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:32:12.058048 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18282
I20260619 06:32:12.058200 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:32:12.058624 389 external_mini_cluster.cc:1428] Setting key 45f90c5877f2f798b2eb8f530eac61cf
I20260619 06:32:12.061302 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46317
--builtin_ntp_servers=127.0.97.84:33949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:32:12.066740 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.056303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:34479 (local address 127.0.97.126:46317)
0619 06:32:12.056474 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.056478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.057852 (+ 1374us) server_negotiation.cc:407] Connection header received
0619 06:32:12.059821 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.059827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.059915 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.060027 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:12.061099 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.062203 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.063700 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.064402 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.064538 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.065820 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:12.065829 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:12.065837 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:12.065860 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:12.065923 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:12.065942 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:12.066040 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.066242 (+ 202us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.066350 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:32:12.067432 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.056819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46317 (local address 127.0.97.66:34479)
0619 06:32:12.057675 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.057750 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:12.059575 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:12.060262 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:12.060271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:12.060347 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:12.060893 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.060915 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.062358 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.062362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.063561 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.063573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.064585 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.064590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.064773 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.065536 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:12.065539 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:12.065574 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:12.065639 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:12.066043 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:12.066051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:12.066056 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:32:12.066187 (+ 131us) client_negotiation.cc:240] Negotiation successful
0619 06:32:12.066448 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":316,"threads_started":1}
I20260619 06:32:12.068801 18409 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46317
I20260619 06:32:12.069136 18409 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:12.069813 18409 heartbeater.cc:511] Master 127.0.97.126:46317 requested a full tablet report, sending...
I20260619 06:32:12.071192 18092 ts_manager.cc:194] Registered new tserver with Master: e927f1b5cc7345b7a95f15ee59bbbbcd (127.0.97.66:43143)
I20260619 06:32:12.072273 18092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:34479
W20260619 06:32:12.383807 18412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:12.384109 18412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:12.384269 18412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:12.391335 18412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:12.391461 18412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:12.391633 18412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:32:12.400432 18412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:33949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:46317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:12.401662 18412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:12.403393 18412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:12.415997 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:12.416117 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:12.416271 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:12.417177 18412 server_base.cc:1034] running on GCE node
I20260619 06:32:12.417739 18412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:12.418617 18412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:12.419859 18412 hybrid_clock.cc:648] HybridClock initialized: now 1781850732419794 us; error 54 us; skew 500 ppm
I20260619 06:32:12.420177 18412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:12.422504 18412 webserver.cc:460] Webserver started at http://127.0.97.67:44241/ using document root <none> and password file <none>
I20260619 06:32:12.423246 18412 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:12.423364 18412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:12.423682 18412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:12.426425 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "ed0b9830e3c44eff82848a82ffed0fbe"
format_stamp: "Formatted at 2026-06-19 06:32:12 on dist-test-slave-4t1f"
server_key: "bfeb3f9893a54348697bc4f3197b9634"
server_key_iv: "5a1c6dce61dbe1f6fd9154e94faf1174"
server_key_version: "encryptionkey@0"
I20260619 06:32:12.427181 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "ed0b9830e3c44eff82848a82ffed0fbe"
format_stamp: "Formatted at 2026-06-19 06:32:12 on dist-test-slave-4t1f"
server_key: "bfeb3f9893a54348697bc4f3197b9634"
server_key_iv: "5a1c6dce61dbe1f6fd9154e94faf1174"
server_key_version: "encryptionkey@0"
I20260619 06:32:12.431918 18412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:32:12.435040 18426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:12.436182 18412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:32:12.436348 18412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "ed0b9830e3c44eff82848a82ffed0fbe"
format_stamp: "Formatted at 2026-06-19 06:32:12 on dist-test-slave-4t1f"
server_key: "bfeb3f9893a54348697bc4f3197b9634"
server_key_iv: "5a1c6dce61dbe1f6fd9154e94faf1174"
server_key_version: "encryptionkey@0"
I20260619 06:32:12.436553 18412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:12.454219 18412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:12.454957 18412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:12.455231 18412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:12.456128 18412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:12.457540 18412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:12.457667 18412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:12.457815 18412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:12.457906 18412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:12.491660 18412 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:37805
I20260619 06:32:12.491712 18538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:37805 every 8 connection(s)
I20260619 06:32:12.492961 18412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:32:12.493414 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18412
I20260619 06:32:12.493597 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:32:12.494189 389 external_mini_cluster.cc:1428] Setting key 95c115b2b98f69624351eed93351bc1e
I20260619 06:32:12.505311 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.495772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35469 (local address 127.0.97.126:46317)
0619 06:32:12.495934 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.495939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.497821 (+ 1882us) server_negotiation.cc:407] Connection header received
0619 06:32:12.499541 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.499545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.499605 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.499687 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:12.500622 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.501411 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.502881 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.503278 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.503372 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.504615 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:12.504619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:12.504623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:12.504638 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:12.504670 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:12.504681 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:12.504757 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.504957 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.505031 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:32:12.506266 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.496518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:46317 (local address 127.0.97.67:35469)
0619 06:32:12.497690 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.497726 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:12.499310 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:12.499883 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:12.499890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:12.499971 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:12.500491 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.500506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.501530 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.501534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.502759 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.502771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.503487 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.503492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.503687 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.504338 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:12.504341 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:12.504377 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:12.504452 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:12.504771 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:12.504778 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:12.504783 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 06:32:12.504929 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 06:32:12.505273 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":346,"threads_started":1}
I20260619 06:32:12.507608 18539 heartbeater.cc:347] Connected to a master server at 127.0.97.126:46317
I20260619 06:32:12.507930 18539 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:12.508849 18539 heartbeater.cc:511] Master 127.0.97.126:46317 requested a full tablet report, sending...
I20260619 06:32:12.510303 18092 ts_manager.cc:194] Registered new tserver with Master: ed0b9830e3c44eff82848a82ffed0fbe (127.0.97.67:37805)
I20260619 06:32:12.511114 18092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:35469
I20260619 06:32:12.521394 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:32:12.531075 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.525287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.0.97.126:46317)
0619 06:32:12.525477 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.525485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.525858 (+ 373us) server_negotiation.cc:407] Connection header received
0619 06:32:12.526041 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.526046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.526124 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.526226 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:12.526693 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.527390 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.528683 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.529187 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.529316 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.530397 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:12.530403 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:12.530409 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:12.530429 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:12.530457 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:12.530468 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:12.530561 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.530699 (+ 138us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.530789 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:32:12.534650 18092 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:32:12.541524 18092 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:32:12.563086 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.552460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50726)
0619 06:32:12.553354 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.553399 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:12.553584 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:12.554981 (+ 1397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:12.554990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:12.555021 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:12.555463 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.555475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.557570 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.557578 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.560209 (+ 2631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.560224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.562105 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.562111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.562270 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.562325 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:32:12.562422 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:32:12.562610 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":150,"thread_start_us":1410,"threads_started":1}
I20260619 06:32:12.563131 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.552616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37805 (local address 127.0.0.1:57090)
0619 06:32:12.554124 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.554144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:12.554281 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:12.555235 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:12.555239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:12.555261 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:12.555826 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.555838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.557687 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.557692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.560530 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.560542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.561912 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.561918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.562448 (+ 530us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.562491 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 06:32:12.562582 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 06:32:12.562719 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"mutex_wait_us":448,"thread_start_us":344,"threads_started":1}
I20260619 06:32:12.563820 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.553251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.0.97.67:37805)
0619 06:32:12.554048 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.554063 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.554408 (+ 345us) server_negotiation.cc:407] Connection header received
0619 06:32:12.554508 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.554516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.554755 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.554945 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:12.556051 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.557536 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.560730 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.561734 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.561886 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.562156 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.563385 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.563544 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":419,"threads_started":1}
I20260619 06:32:12.564419 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.552640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.0.97.65:36889)
0619 06:32:12.554467 (+ 1827us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.554481 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.554503 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:32:12.554579 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.554585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.554763 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.555558 (+ 795us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:12.555824 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.557415 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.560417 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.561898 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.563637 (+ 1739us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.563875 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.563937 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.564090 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1674,"thread_start_us":1544,"threads_started":1}
I20260619 06:32:12.567636 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.554275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43143 (local address 127.0.0.1:34308)
0619 06:32:12.559479 (+ 5204us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.559496 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:12.559610 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:12.560279 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:12.560283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:12.560304 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:12.560758 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.560772 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.562622 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.562627 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.564725 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.564739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.566851 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.566856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.567043 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.567065 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:32:12.567166 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:32:12.567281 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5097,"thread_start_us":4954,"threads_started":1}
I20260619 06:32:12.567814 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.554916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34308 (local address 127.0.97.66:43143)
0619 06:32:12.559123 (+ 4207us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.559131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.559725 (+ 594us) server_negotiation.cc:407] Connection header received
0619 06:32:12.559831 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.559842 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.560067 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.560280 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:12.560930 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.562467 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.564933 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.566353 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.566536 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.566780 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.567222 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.567368 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4057,"thread_start_us":3926,"threads_started":1}
I20260619 06:32:12.567530 18474 tablet_service.cc:1467] Processing CreateTablet for tablet aaea809a4df64e05a41636ac9db0bb5d (DEFAULT_TABLE table=test-table [id=d73a7baf59d249b3854e7aaf37d6ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:12.567772 18213 tablet_service.cc:1467] Processing CreateTablet for tablet aaea809a4df64e05a41636ac9db0bb5d (DEFAULT_TABLE table=test-table [id=d73a7baf59d249b3854e7aaf37d6ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:12.569015 18474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaea809a4df64e05a41636ac9db0bb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:12.569015 18213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaea809a4df64e05a41636ac9db0bb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:12.573387 18344 tablet_service.cc:1467] Processing CreateTablet for tablet aaea809a4df64e05a41636ac9db0bb5d (DEFAULT_TABLE table=test-table [id=d73a7baf59d249b3854e7aaf37d6ba1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:12.574455 18344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aaea809a4df64e05a41636ac9db0bb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:12.583590 18557 tablet_bootstrap.cc:492] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: Bootstrap starting.
I20260619 06:32:12.584335 18558 tablet_bootstrap.cc:492] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Bootstrap starting.
I20260619 06:32:12.586503 18559 tablet_bootstrap.cc:492] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Bootstrap starting.
I20260619 06:32:12.587980 18558 tablet_bootstrap.cc:654] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:12.588420 18557 tablet_bootstrap.cc:654] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:12.589210 18558 log.cc:824] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:12.589632 18557 log.cc:824] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:12.589819 18559 tablet_bootstrap.cc:654] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:12.591125 18559 log.cc:824] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:12.594141 18558 tablet_bootstrap.cc:492] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: No bootstrap required, opened a new log
I20260619 06:32:12.594340 18559 tablet_bootstrap.cc:492] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: No bootstrap required, opened a new log
I20260619 06:32:12.594453 18557 tablet_bootstrap.cc:492] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: No bootstrap required, opened a new log
I20260619 06:32:12.594565 18558 ts_tablet_manager.cc:1397] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20260619 06:32:12.594589 18559 ts_tablet_manager.cc:1397] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:32:12.594657 18557 ts_tablet_manager.cc:1397] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260619 06:32:12.601873 18558 raft_consensus.cc:348] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.602231 18558 raft_consensus.cc:374] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:12.602317 18558 raft_consensus.cc:729] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed0b9830e3c44eff82848a82ffed0fbe, State: Initialized, Role: FOLLOWER
I20260619 06:32:12.602907 18558 consensus_queue.cc:260] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.603932 18539 heartbeater.cc:503] Master 127.0.97.126:46317 was elected leader, sending a full tablet report...
I20260619 06:32:12.604564 18558 ts_tablet_manager.cc:1428] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260619 06:32:12.605139 18559 raft_consensus.cc:348] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.605139 18557 raft_consensus.cc:348] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.605544 18557 raft_consensus.cc:374] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:12.605544 18559 raft_consensus.cc:374] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:12.605644 18559 raft_consensus.cc:729] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e927f1b5cc7345b7a95f15ee59bbbbcd, State: Initialized, Role: FOLLOWER
I20260619 06:32:12.605645 18557 raft_consensus.cc:729] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c875dd2c1d7474b992cf76a8d0c6bb1, State: Initialized, Role: FOLLOWER
I20260619 06:32:12.606194 18557 consensus_queue.cc:260] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.606195 18559 consensus_queue.cc:260] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.613344 18409 heartbeater.cc:503] Master 127.0.97.126:46317 was elected leader, sending a full tablet report...
I20260619 06:32:12.614252 18278 heartbeater.cc:503] Master 127.0.97.126:46317 was elected leader, sending a full tablet report...
I20260619 06:32:12.614423 18559 ts_tablet_manager.cc:1428] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Time spent starting tablet: real 0.020s user 0.016s sys 0.005s
I20260619 06:32:12.616932 18557 ts_tablet_manager.cc:1428] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: Time spent starting tablet: real 0.022s user 0.017s sys 0.004s
W20260619 06:32:12.628836 18279 tablet.cc:2369] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:12.721992 18564 raft_consensus.cc:482] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:32:12.722373 18564 raft_consensus.cc:504] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.724367 18564 leader_election.cc:288] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed0b9830e3c44eff82848a82ffed0fbe (127.0.97.67:37805), e927f1b5cc7345b7a95f15ee59bbbbcd (127.0.97.66:43143)
I20260619 06:32:12.731534 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.724791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:57547 (local address 127.0.97.66:43143)
0619 06:32:12.725006 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.725017 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.725793 (+ 776us) server_negotiation.cc:407] Connection header received
0619 06:32:12.726073 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.726115 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.726191 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.726259 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:12.726944 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.727695 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.729883 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.730606 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.730696 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.730818 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.731198 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.731316 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:32:12.731563 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.724916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:52149 (local address 127.0.97.67:37805)
0619 06:32:12.725172 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.725181 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.725971 (+ 790us) server_negotiation.cc:407] Connection header received
0619 06:32:12.726139 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.726144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.726206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.726327 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:12.727027 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.727889 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.729796 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.730628 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.730788 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.730889 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.731155 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.731286 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:32:12.731886 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.724770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:37805 (local address 127.0.97.65:52149)
0619 06:32:12.725862 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.725875 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:12.726009 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:12.726359 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:12.726362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:12.726389 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:12.726824 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.726833 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.728040 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.728043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.729606 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.729618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.730847 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.730854 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.730996 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.731018 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:32:12.731093 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 06:32:12.731230 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":46,"thread_start_us":474,"threads_started":1}
I20260619 06:32:12.732013 18567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.724750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43143 (local address 127.0.97.65:57547)
0619 06:32:12.725672 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.725690 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:12.725885 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:12.726314 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:12.726319 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:12.726345 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:12.726788 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.726801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.727837 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.727843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.729679 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:12.729692 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.730793 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.730799 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:12.731011 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.731036 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 06:32:12.731157 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 06:32:12.731285 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":100,"thread_start_us":825,"threads_started":1}
I20260619 06:32:12.732861 18494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea809a4df64e05a41636ac9db0bb5d" candidate_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" is_pre_election: true
I20260619 06:32:12.732905 18364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea809a4df64e05a41636ac9db0bb5d" candidate_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" is_pre_election: true
I20260619 06:32:12.733397 18494 raft_consensus.cc:2454] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c875dd2c1d7474b992cf76a8d0c6bb1 in term 0.
I20260619 06:32:12.733492 18364 raft_consensus.cc:2454] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c875dd2c1d7474b992cf76a8d0c6bb1 in term 0.
I20260619 06:32:12.734091 18166 leader_election.cc:302] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c875dd2c1d7474b992cf76a8d0c6bb1, ed0b9830e3c44eff82848a82ffed0fbe; no voters:
I20260619 06:32:12.734515 18564 raft_consensus.cc:2781] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:32:12.734730 18564 raft_consensus.cc:482] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:32:12.734874 18564 raft_consensus.cc:3037] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:12.737236 18564 raft_consensus.cc:504] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
I20260619 06:32:12.737928 18564 leader_election.cc:288] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [CANDIDATE]: Term 1 election: Requested vote from peers ed0b9830e3c44eff82848a82ffed0fbe (127.0.97.67:37805), e927f1b5cc7345b7a95f15ee59bbbbcd (127.0.97.66:43143)
I20260619 06:32:12.738333 18494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea809a4df64e05a41636ac9db0bb5d" candidate_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed0b9830e3c44eff82848a82ffed0fbe"
I20260619 06:32:12.738425 18364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aaea809a4df64e05a41636ac9db0bb5d" candidate_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd"
I20260619 06:32:12.738571 18494 raft_consensus.cc:3037] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:12.738613 18364 raft_consensus.cc:3037] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:12.742085 18494 raft_consensus.cc:2454] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c875dd2c1d7474b992cf76a8d0c6bb1 in term 1.
I20260619 06:32:12.742101 18364 raft_consensus.cc:2454] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c875dd2c1d7474b992cf76a8d0c6bb1 in term 1.
I20260619 06:32:12.742537 18169 leader_election.cc:302] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c875dd2c1d7474b992cf76a8d0c6bb1, e927f1b5cc7345b7a95f15ee59bbbbcd; no voters:
I20260619 06:32:12.742843 18564 raft_consensus.cc:2781] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:12.743268 18564 raft_consensus.cc:686] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 1 LEADER]: Becoming Leader. State: Replica: 4c875dd2c1d7474b992cf76a8d0c6bb1, State: Running, Role: LEADER
I20260619 06:32:12.743702 18564 consensus_queue.cc:237] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } }
W20260619 06:32:12.744963 18540 tablet.cc:2369] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:12.748346 18092 catalog_manager.cc:5495] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c875dd2c1d7474b992cf76a8d0c6bb1 (127.0.97.65). New cstate: current_term: 1 leader_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c875dd2c1d7474b992cf76a8d0c6bb1" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 } health_report { overall_health: UNKNOWN } } }
I20260619 06:32:12.804832 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.799409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.0.97.65:36889)
0619 06:32:12.799639 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.799646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.799664 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:32:12.799917 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.799921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.800007 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.800160 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:32:12.800521 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.801246 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.802761 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.803175 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.803310 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.803793 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:32:12.804259 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:32:12.804427 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.804478 (+ 51us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.804552 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260619 06:32:12.805787 18410 tablet.cc:2369] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:12.812930 18364 raft_consensus.cc:1261] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 1 FOLLOWER]: Refusing update from remote peer 4c875dd2c1d7474b992cf76a8d0c6bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:12.812953 18494 raft_consensus.cc:1261] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 1 FOLLOWER]: Refusing update from remote peer 4c875dd2c1d7474b992cf76a8d0c6bb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:12.813845 18568 consensus_queue.cc:1035] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed0b9830e3c44eff82848a82ffed0fbe" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 37805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:12.814263 18564 consensus_queue.cc:1035] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e927f1b5cc7345b7a95f15ee59bbbbcd" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:12.841502 18091 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:12.841825 18091 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:12.845474 18091 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7547017c9a340cc8f8166943602b46c: Sending DeleteTablet for 3 replicas of tablet aaea809a4df64e05a41636ac9db0bb5d
I20260619 06:32:12.846575 18213 tablet_service.cc:1514] Processing DeleteTablet for tablet aaea809a4df64e05a41636ac9db0bb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:12 UTC) from {username='slave'} at 127.0.0.1:50726
I20260619 06:32:12.846692 18344 tablet_service.cc:1514] Processing DeleteTablet for tablet aaea809a4df64e05a41636ac9db0bb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:12 UTC) from {username='slave'} at 127.0.0.1:34308
I20260619 06:32:12.847246 18474 tablet_service.cc:1514] Processing DeleteTablet for tablet aaea809a4df64e05a41636ac9db0bb5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:12 UTC) from {username='slave'} at 127.0.0.1:57090
I20260619 06:32:12.847431 18581 tablet_replica.cc:330] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: stopping tablet replica
I20260619 06:32:12.847832 18581 raft_consensus.cc:2229] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:32:12.848433 18581 raft_consensus.cc:2258] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:12.848686 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18151
I20260619 06:32:12.858592 18581 ts_tablet_manager.cc:1905] T aaea809a4df64e05a41636ac9db0bb5d P 4c875dd2c1d7474b992cf76a8d0c6bb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:12.861898 18582 tablet_replica.cc:330] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: stopping tablet replica
I20260619 06:32:12.862290 18582 raft_consensus.cc:2229] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:12.862752 18582 raft_consensus.cc:2258] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:12.864864 18582 ts_tablet_manager.cc:1905] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:12.865880 18583 tablet_replica.cc:330] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: stopping tablet replica
I20260619 06:32:12.866356 18583 raft_consensus.cc:2229] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:12.866840 18583 raft_consensus.cc:2258] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:12.870621 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.849664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.0.97.65:36889)
0619 06:32:12.849834 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.849840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.861729 (+ 11889us) server_negotiation.cc:407] Connection header received
0619 06:32:12.862171 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.862180 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.862323 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.862412 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:12.863201 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.864518 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.866552 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.867210 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.867300 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.868843 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:12.868853 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:12.868872 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:12.869653 (+ 781us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:12.869710 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:12.869734 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:12.869842 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.870095 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.870199 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:32:12.871491 18582 ts_tablet_manager.cc:1918] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:12.871665 18582 log.cc:1196] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/aaea809a4df64e05a41636ac9db0bb5d
I20260619 06:32:12.871752 18273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:12.872009 18583 ts_tablet_manager.cc:1905] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:12.872278 18582 ts_tablet_manager.cc:1939] T aaea809a4df64e05a41636ac9db0bb5d P e927f1b5cc7345b7a95f15ee59bbbbcd: Deleting consensus metadata
I20260619 06:32:12.874329 18079 catalog_manager.cc:4841] TS e927f1b5cc7345b7a95f15ee59bbbbcd (127.0.97.66:43143): tablet aaea809a4df64e05a41636ac9db0bb5d (table test-table [id=d73a7baf59d249b3854e7aaf37d6ba1a]) successfully deleted
I20260619 06:32:12.878491 18583 ts_tablet_manager.cc:1918] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:12.878647 18583 log.cc:1196] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/aaea809a4df64e05a41636ac9db0bb5d
I20260619 06:32:12.879087 18583 ts_tablet_manager.cc:1939] T aaea809a4df64e05a41636ac9db0bb5d P ed0b9830e3c44eff82848a82ffed0fbe: Deleting consensus metadata
I20260619 06:32:12.880581 18076 catalog_manager.cc:4841] TS ed0b9830e3c44eff82848a82ffed0fbe (127.0.97.67:37805): tablet aaea809a4df64e05a41636ac9db0bb5d (table test-table [id=d73a7baf59d249b3854e7aaf37d6ba1a]) successfully deleted
I20260619 06:32:12.969946 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18151
W20260619 06:32:12.987599 18079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:36889 (error 108)
I20260619 06:32:12.988322 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18282
I20260619 06:32:12.989638 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.989093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50746)
0619 06:32:12.989295 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:12.989451 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:32:12.989888 18079 catalog_manager.cc:4584] TS 4c875dd2c1d7474b992cf76a8d0c6bb1 (127.0.97.65:36889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aaea809a4df64e05a41636ac9db0bb5d: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
I20260619 06:32:12.995260 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:12.989185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34320 (local address 127.0.97.66:43143)
0619 06:32:12.989347 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:12.989356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:12.989431 (+ 75us) server_negotiation.cc:407] Connection header received
0619 06:32:12.989581 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:12.989587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:12.989701 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:12.989821 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:12.990303 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.991089 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.992304 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:12.992686 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:12.992782 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:12.993880 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:12.993890 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:12.993903 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:12.994490 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:12.994554 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:12.994575 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:12.994673 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:12.994762 (+ 89us) server_negotiation.cc:299] Negotiation successful
0619 06:32:12.994844 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:32:12.995872 18404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:13.043496 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:13.043029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50748)
0619 06:32:13.043241 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:13.043324 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 06:32:13.044525 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:13.044188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50762)
0619 06:32:13.044334 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:13.044407 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260619 06:32:13.084661 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18282
I20260619 06:32:13.101150 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18412
I20260619 06:32:13.104929 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:13.104478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50774)
0619 06:32:13.104653 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:13.104736 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:32:13.105854 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:13.105531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50786)
0619 06:32:13.105672 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:13.105746 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:32:13.108024 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:13.101902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.0.97.67:37805)
0619 06:32:13.102067 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:13.102071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:13.102089 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:32:13.102235 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:13.102239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:13.102321 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:13.102405 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:13.102754 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:13.103550 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:13.104785 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:13.105420 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:13.105557 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:13.106728 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:13.106737 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:13.106753 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:13.107322 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:13.107367 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:13.107385 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:13.107506 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:13.107636 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 06:32:13.107744 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:32:13.108595 18534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:13.186374 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:13.185908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50800)
0619 06:32:13.186114 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:13.186195 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 06:32:13.187494 18555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:13.187085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36889 (local address 127.0.0.1:50810)
0619 06:32:13.187315 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:13.187391 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:36889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260619 06:32:13.195477 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18412
I20260619 06:32:13.211650 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18061
I20260619 06:32:13.212657 18122 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:13.319797 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18061
2026-06-19T06:32:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2744 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:13 dist-test-slave-4t1f krb5kdc[18592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:13 dist-test-slave-4t1f krb5kdc[18592](info): set up 2 sockets
Jun 19 06:32:13 dist-test-slave-4t1f krb5kdc[18592](info): commencing operation
krb5kdc: starting...
W20260619 06:32:14.408939 389 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:14 dist-test-slave-4t1f krb5kdc[18592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:14.642289 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45269
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:14.954980 18605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:14.955273 18605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:14.955361 18605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:14.962298 18605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:14.962386 18605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:14.962430 18605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:14.962486 18605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:14.962527 18605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:14.971680 18605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:14.972826 18605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:14.974067 18605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:14.985200 18613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:14.985175 18610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:14.985160 18611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:14.985911 18605 server_base.cc:1034] running on GCE node
I20260619 06:32:14.986660 18605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:14.987969 18605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:14.989176 18605 hybrid_clock.cc:648] HybridClock initialized: now 1781850734989125 us; error 59 us; skew 500 ppm
I20260619 06:32:14.989506 18605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:14 dist-test-slave-4t1f krb5kdc[18592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850734, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:14.992491 18605 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:14.993882 18605 webserver.cc:460] Webserver started at http://127.0.97.126:35137/ using document root <none> and password file <none>
I20260619 06:32:14.994485 18605 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:14.994577 18605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:14.994812 18605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:14.997682 18605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "e5e58deceaec42bab7b5bee6fac6f70f"
format_stamp: "Formatted at 2026-06-19 06:32:14 on dist-test-slave-4t1f"
server_key: "f2128352fc77d750348b531a6f10767f"
server_key_iv: "cc27cd1f2930a864f9ec3fcdaff2ac45"
server_key_version: "encryptionkey@0"
I20260619 06:32:14.998343 18605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "e5e58deceaec42bab7b5bee6fac6f70f"
format_stamp: "Formatted at 2026-06-19 06:32:14 on dist-test-slave-4t1f"
server_key: "f2128352fc77d750348b531a6f10767f"
server_key_iv: "cc27cd1f2930a864f9ec3fcdaff2ac45"
server_key_version: "encryptionkey@0"
I20260619 06:32:15.003002 18605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:32:15.006156 18620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:15.007359 18605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:32:15.007484 18605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "e5e58deceaec42bab7b5bee6fac6f70f"
format_stamp: "Formatted at 2026-06-19 06:32:14 on dist-test-slave-4t1f"
server_key: "f2128352fc77d750348b531a6f10767f"
server_key_iv: "cc27cd1f2930a864f9ec3fcdaff2ac45"
server_key_version: "encryptionkey@0"
I20260619 06:32:15.007629 18605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:15.040953 18605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:15.045286 18605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:15.045511 18605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:15.063702 18605 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45269
I20260619 06:32:15.063724 18671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45269 every 8 connection(s)
I20260619 06:32:15.065101 18605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:15.068656 18672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:15.073892 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18605
I20260619 06:32:15.074080 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:15.074507 389 external_mini_cluster.cc:1428] Setting key d838a978d65dfd7a1ea17930453a5c55
I20260619 06:32:15.079286 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f: Bootstrap starting.
I20260619 06:32:15.082845 18672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:15.084069 18672 log.cc:824] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:15.087348 18672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f: No bootstrap required, opened a new log
Jun 19 06:32:15 dist-test-slave-4t1f krb5kdc[18592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:15.095085 18672 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45269 } }
I20260619 06:32:15.095482 18672 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:15.095611 18672 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e58deceaec42bab7b5bee6fac6f70f, State: Initialized, Role: FOLLOWER
I20260619 06:32:15.096350 18672 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45269 } }
I20260619 06:32:15.096657 18672 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:15.096825 18672 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:15.097023 18672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:15.097884 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:15.076302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.0.97.126:45269)
0619 06:32:15.077046 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:15.077102 (+ 56us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:15.077158 (+ 56us) server_negotiation.cc:407] Connection header received
0619 06:32:15.078835 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:15.078854 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:15.079203 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:15.079552 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:15.081380 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:15.082611 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:15.084480 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:15.085602 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:15.085727 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:15.090488 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:15.090496 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:15.090509 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:15.090550 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:15.093743 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:15.094451 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:15.094457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:15.094463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:15.094615 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:15.094989 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:15.094994 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:15.094997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:15.095556 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:15.095889 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:15.096292 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 06:32:15.096648 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":323,"threads_started":1}
I20260619 06:32:15.099920 18672 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45269 } }
I20260619 06:32:15.100431 18672 leader_election.cc:302] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5e58deceaec42bab7b5bee6fac6f70f; no voters:
I20260619 06:32:15.101019 18672 leader_election.cc:288] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:15.101172 18677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:15.101881 18677 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [term 1 LEADER]: Becoming Leader. State: Replica: e5e58deceaec42bab7b5bee6fac6f70f, State: Running, Role: LEADER
I20260619 06:32:15.102461 18677 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45269 } }
I20260619 06:32:15.103036 18672 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:15.105218 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e58deceaec42bab7b5bee6fac6f70f. Latest consensus state: current_term: 1 leader_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45269 } } }
I20260619 06:32:15.105649 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:15.106006 18678 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5e58deceaec42bab7b5bee6fac6f70f" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45269 } } }
I20260619 06:32:15.106282 18678 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:15.110587 18686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:15.115265 18686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:15.123378 18686 catalog_manager.cc:1347] Generated new cluster ID: 165dd0e388a342d39d014c3a0d144a4d
I20260619 06:32:15.123493 18686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:15.135779 18686 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:15.136996 18686 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:15.145442 18686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f: Generated new TSK 0
I20260619 06:32:15.146241 18686 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:32:15.206642 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45269
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:15.531036 18698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:15.531394 18698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:15.531584 18698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:15.539439 18698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:15.539583 18698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:15.539788 18698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:32:15.550084 18698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:15.551307 18698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:15.552625 18698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:15.565670 18703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:15.565626 18704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:15.565644 18706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:15.566428 18698 server_base.cc:1034] running on GCE node
I20260619 06:32:15.567018 18698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:15.567857 18698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:15.569048 18698 hybrid_clock.cc:648] HybridClock initialized: now 1781850735568996 us; error 47 us; skew 500 ppm
I20260619 06:32:15.569411 18698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:15 dist-test-slave-4t1f krb5kdc[18592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:15.572454 18698 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:32:15.573876 18698 webserver.cc:460] Webserver started at http://127.0.97.65:44633/ using document root <none> and password file <none>
I20260619 06:32:15.574522 18698 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:15.574659 18698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:15.574951 18698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:15.577731 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "d2ec66c7510440378a1646aaac2d7f0b"
format_stamp: "Formatted at 2026-06-19 06:32:15 on dist-test-slave-4t1f"
server_key: "7908580d6eba6ef6a8b1c208c6c01501"
server_key_iv: "7497eef6cd6d3958bda6f949232ce9c4"
server_key_version: "encryptionkey@0"
I20260619 06:32:15.578476 18698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "d2ec66c7510440378a1646aaac2d7f0b"
format_stamp: "Formatted at 2026-06-19 06:32:15 on dist-test-slave-4t1f"
server_key: "7908580d6eba6ef6a8b1c208c6c01501"
server_key_iv: "7497eef6cd6d3958bda6f949232ce9c4"
server_key_version: "encryptionkey@0"
I20260619 06:32:15.584318 18698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:32:15.587575 18713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:15.588857 18698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:32:15.589025 18698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "d2ec66c7510440378a1646aaac2d7f0b"
format_stamp: "Formatted at 2026-06-19 06:32:15 on dist-test-slave-4t1f"
server_key: "7908580d6eba6ef6a8b1c208c6c01501"
server_key_iv: "7497eef6cd6d3958bda6f949232ce9c4"
server_key_version: "encryptionkey@0"
I20260619 06:32:15.589291 18698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:15.606987 18698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:15.610852 18698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:15.611143 18698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:15.612109 18698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:15.613426 18698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:15.613523 18698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:15.613668 18698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:15.613732 18698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:15.644274 18698 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:42277
I20260619 06:32:15.644407 18825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:42277 every 8 connection(s)
I20260619 06:32:15.645756 18698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:32:15.651180 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18698
I20260619 06:32:15.651358 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:32:15.651932 389 external_mini_cluster.cc:1428] Setting key 53227227449044dc829be822ecea3f2b
Jun 19 06:32:15 dist-test-slave-4t1f krb5kdc[18592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:15.667805 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:15.648159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:49527 (local address 127.0.97.126:45269)
0619 06:32:15.648528 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:15.648532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:15.649598 (+ 1066us) server_negotiation.cc:407] Connection header received
0619 06:32:15.651645 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:15.651649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:15.651724 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:15.651813 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:15.653630 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:15.654484 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:15.656958 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:15.657417 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:15.657516 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:15.661897 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:15.661906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:15.661914 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:15.661958 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:15.664597 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:15.665342 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:15.665351 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:15.665357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:15.665504 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:15.665912 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:15.665920 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:15.665926 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:15.666290 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:15.666437 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:15.667079 (+ 642us) server_negotiation.cc:299] Negotiation successful
0619 06:32:15.667344 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":188,"threads_started":1}
I20260619 06:32:15.668339 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:15.648740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45269 (local address 127.0.97.65:49527)
0619 06:32:15.649447 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:15.649484 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:15.651361 (+ 1877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:15.652106 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:15.652116 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:15.652759 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:15.653466 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:15.653483 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:15.654651 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:15.654660 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:15.656804 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:15.656821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:15.657628 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:15.657637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:15.657956 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:15.658662 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:15.658709 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:15.661672 (+ 2963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:15.664726 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:15.664731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:15.664742 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:15.665211 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:15.665619 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:15.665622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:15.665624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:15.665782 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:15.666405 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:15.666410 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:15.666690 (+ 280us) client_negotiation.cc:769] Sending connection context
0619 06:32:15.666915 (+ 225us) client_negotiation.cc:240] Negotiation successful
0619 06:32:15.667281 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":306,"threads_started":1}
I20260619 06:32:15.669886 18826 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45269
I20260619 06:32:15.670188 18826 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:15.670987 18826 heartbeater.cc:511] Master 127.0.97.126:45269 requested a full tablet report, sending...
I20260619 06:32:15.673300 18637 ts_manager.cc:194] Registered new tserver with Master: d2ec66c7510440378a1646aaac2d7f0b (127.0.97.65:42277)
I20260619 06:32:15.675495 18637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:49527
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:32:15.708163 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45269
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:16.019593 18832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:16.019843 18832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:16.019997 18832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:16.026914 18832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:16.027019 18832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:16.027184 18832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:32:16.036310 18832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:16.037493 18832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:16.038656 18832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:16.050901 18838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:16.050972 18837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:16.050976 18840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:16.051846 18832 server_base.cc:1034] running on GCE node
I20260619 06:32:16.052408 18832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:16.053305 18832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:16.054522 18832 hybrid_clock.cc:648] HybridClock initialized: now 1781850736054486 us; error 47 us; skew 500 ppm
I20260619 06:32:16.054808 18832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:16 dist-test-slave-4t1f krb5kdc[18592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:16.060609 18832 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:32:16.061985 18832 webserver.cc:460] Webserver started at http://127.0.97.66:46253/ using document root <none> and password file <none>
I20260619 06:32:16.062601 18832 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:16.062743 18832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:16.063028 18832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:16.065829 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "04fb6ca537454e63b293ff5c41f99cb2"
format_stamp: "Formatted at 2026-06-19 06:32:16 on dist-test-slave-4t1f"
server_key: "23e1248ec3965992c48c882a1aeb74cd"
server_key_iv: "e4982d23e5d251af875f18a3825cfbc5"
server_key_version: "encryptionkey@0"
I20260619 06:32:16.066558 18832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "04fb6ca537454e63b293ff5c41f99cb2"
format_stamp: "Formatted at 2026-06-19 06:32:16 on dist-test-slave-4t1f"
server_key: "23e1248ec3965992c48c882a1aeb74cd"
server_key_iv: "e4982d23e5d251af875f18a3825cfbc5"
server_key_version: "encryptionkey@0"
I20260619 06:32:16.071441 18832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260619 06:32:16.078199 18847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:16.079856 18832 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260619 06:32:16.080014 18832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "04fb6ca537454e63b293ff5c41f99cb2"
format_stamp: "Formatted at 2026-06-19 06:32:16 on dist-test-slave-4t1f"
server_key: "23e1248ec3965992c48c882a1aeb74cd"
server_key_iv: "e4982d23e5d251af875f18a3825cfbc5"
server_key_version: "encryptionkey@0"
I20260619 06:32:16.080292 18832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:16.100716 18832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:16.104408 18832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:16.104625 18832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:16.105580 18832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:16.106822 18832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:16.106889 18832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:16.106967 18832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:16.107009 18832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:16.138931 18832 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43325
I20260619 06:32:16.139016 18959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43325 every 8 connection(s)
I20260619 06:32:16.140167 18832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:32:16.148479 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18832
I20260619 06:32:16.148669 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:32:16.149260 389 external_mini_cluster.cc:1428] Setting key 09cb0ea4e9bc73b8eea6a20030c15ee7
Jun 19 06:32:16 dist-test-slave-4t1f krb5kdc[18592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:16.160887 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.142965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:52943 (local address 127.0.97.126:45269)
0619 06:32:16.143151 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:16.143156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:16.144231 (+ 1075us) server_negotiation.cc:407] Connection header received
0619 06:32:16.146090 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:16.146096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:16.146193 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:16.146302 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:16.147765 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.148493 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.150469 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.151096 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.151232 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.155173 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:16.155178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:16.155181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:16.155209 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:16.157776 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:16.158497 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:16.158505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:16.158511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:16.158686 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:16.159237 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:16.159243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:16.159249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:16.159613 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:16.159773 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:16.160416 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 06:32:16.160613 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:32:16.162235 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.143337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45269 (local address 127.0.97.66:52943)
0619 06:32:16.144053 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:16.144091 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:16.145831 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:16.146523 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:16.146530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:16.146995 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:16.147587 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.147604 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.148690 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.148697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.150319 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.150331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.151331 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.151338 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.151599 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.152271 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:16.152307 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:16.154960 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:16.157901 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:16.157909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:16.157922 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:16.158354 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:16.158822 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:16.158829 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:16.158835 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:16.159047 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:16.159737 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:16.159742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:16.160013 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 06:32:16.160283 (+ 270us) client_negotiation.cc:240] Negotiation successful
0619 06:32:16.160838 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":283,"threads_started":1}
I20260619 06:32:16.163655 18960 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45269
I20260619 06:32:16.163945 18960 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:16.164582 18960 heartbeater.cc:511] Master 127.0.97.126:45269 requested a full tablet report, sending...
I20260619 06:32:16.166010 18637 ts_manager.cc:194] Registered new tserver with Master: 04fb6ca537454e63b293ff5c41f99cb2 (127.0.97.66:43325)
I20260619 06:32:16.167217 18637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:52943
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:32:16.202229 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45269
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:37423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:16.512815 18965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:16.513077 18965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:16.513286 18965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:16.520354 18965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:16.520448 18965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:16.520557 18965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:32:16.529601 18965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:37423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:45269
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:16.531009 18965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:16.532378 18965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:16.544595 18971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:16.544651 18973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:16.545233 18970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:16.545818 18965 server_base.cc:1034] running on GCE node
I20260619 06:32:16.546415 18965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:16.547315 18965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:16.548532 18965 hybrid_clock.cc:648] HybridClock initialized: now 1781850736548486 us; error 58 us; skew 500 ppm
I20260619 06:32:16.548830 18965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:16 dist-test-slave-4t1f krb5kdc[18592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:16.551826 18965 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:32:16.553185 18965 webserver.cc:460] Webserver started at http://127.0.97.67:46075/ using document root <none> and password file <none>
I20260619 06:32:16.553820 18965 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:16.553963 18965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:16.554255 18965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:16.556923 18965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "87711bf110904224a43e101461ae1d20"
format_stamp: "Formatted at 2026-06-19 06:32:16 on dist-test-slave-4t1f"
server_key: "d72c52e76a35784bd9fc143952c62463"
server_key_iv: "ae52f1478507bac912c3b30b9644de0c"
server_key_version: "encryptionkey@0"
I20260619 06:32:16.557763 18965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "87711bf110904224a43e101461ae1d20"
format_stamp: "Formatted at 2026-06-19 06:32:16 on dist-test-slave-4t1f"
server_key: "d72c52e76a35784bd9fc143952c62463"
server_key_iv: "ae52f1478507bac912c3b30b9644de0c"
server_key_version: "encryptionkey@0"
I20260619 06:32:16.562623 18965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 06:32:16.565860 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:16.567054 18965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:32:16.567233 18965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "87711bf110904224a43e101461ae1d20"
format_stamp: "Formatted at 2026-06-19 06:32:16 on dist-test-slave-4t1f"
server_key: "d72c52e76a35784bd9fc143952c62463"
server_key_iv: "ae52f1478507bac912c3b30b9644de0c"
server_key_version: "encryptionkey@0"
I20260619 06:32:16.567420 18965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:16.593595 18965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:16.597491 18965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:16.597710 18965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:16.598560 18965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:16.599746 18965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:16.599817 18965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:16.599893 18965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:16.599936 18965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:16.632462 18965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:38999
I20260619 06:32:16.632541 19092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:38999 every 8 connection(s)
I20260619 06:32:16.633673 18965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:32:16.644114 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 18965
I20260619 06:32:16.644315 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:32:16.644853 389 external_mini_cluster.cc:1428] Setting key fd0678cd401f5261f3d63e1378ec0e49
Jun 19 06:32:16 dist-test-slave-4t1f krb5kdc[18592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:16.653640 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.636066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35121 (local address 127.0.97.126:45269)
0619 06:32:16.636233 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:16.636239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:16.637318 (+ 1079us) server_negotiation.cc:407] Connection header received
0619 06:32:16.639091 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:16.639096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:16.639188 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:16.639282 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:16.640626 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.641609 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.643177 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.643895 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.644037 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.648233 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:16.648239 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:16.648244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:16.648281 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:16.651209 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:16.651844 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:16.651848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:16.651850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:16.651969 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:16.652305 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:16.652308 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:16.652310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:16.652556 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:16.652674 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:16.653265 (+ 591us) server_negotiation.cc:299] Negotiation successful
0619 06:32:16.653417 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:32:16.654501 19095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.636457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45269 (local address 127.0.97.67:35121)
0619 06:32:16.637149 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:16.637193 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:16.638826 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:16.639468 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:16.639476 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:16.639960 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:16.640479 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.640494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.641779 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.641783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.642994 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.643009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.644330 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.644336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.644593 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.645358 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:16.645392 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:16.648044 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:16.651330 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:16.651337 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:16.651347 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:16.651749 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:16.652056 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:16.652059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:16.652062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:16.652217 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:16.652670 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:16.652675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:16.652924 (+ 249us) client_negotiation.cc:769] Sending connection context
0619 06:32:16.653181 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 06:32:16.653499 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":252,"threads_started":1}
I20260619 06:32:16.655751 19093 heartbeater.cc:347] Connected to a master server at 127.0.97.126:45269
I20260619 06:32:16.655997 19093 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:16.656574 19093 heartbeater.cc:511] Master 127.0.97.126:45269 requested a full tablet report, sending...
I20260619 06:32:16.657861 18637 ts_manager.cc:194] Registered new tserver with Master: 87711bf110904224a43e101461ae1d20 (127.0.97.67:38999)
I20260619 06:32:16.658787 18637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:35121
I20260619 06:32:16.660183 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:32:16.675634 18828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.664016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33394 (local address 127.0.97.126:45269)
0619 06:32:16.664194 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:16.664201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:16.664563 (+ 362us) server_negotiation.cc:407] Connection header received
0619 06:32:16.664677 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:16.664681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:16.664731 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:16.664801 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:16.666100 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.666771 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.667957 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.668334 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.668418 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.671014 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:16.671018 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:16.671021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:16.671047 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:16.673351 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:16.673999 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:16.674003 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:16.674005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:16.674116 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:16.674446 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:16.674449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:16.674451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:16.674724 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:16.674907 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:16.675183 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 06:32:16.675297 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 06:32:16.678815 18826 heartbeater.cc:503] Master 127.0.97.126:45269 was elected leader, sending a full tablet report...
I20260619 06:32:16.679452 18637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:32:16.686444 18637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:32:16.711284 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.697893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.0.97.65:42277)
0619 06:32:16.699598 (+ 1705us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:16.699605 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:16.700295 (+ 690us) server_negotiation.cc:407] Connection header received
0619 06:32:16.700595 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:16.700601 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:16.700838 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:16.701014 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:16.702039 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.704102 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.707543 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.709392 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.709490 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.709718 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:16.710609 (+ 891us) server_negotiation.cc:299] Negotiation successful
0619 06:32:16.710816 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1567,"thread_start_us":1408,"threads_started":1}
I20260619 06:32:16.712415 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.697635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42277 (local address 127.0.0.1:33952)
0619 06:32:16.700161 (+ 2526us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:16.700191 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:16.700386 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:16.701343 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:16.701350 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:16.701391 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:16.701827 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.701842 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.704356 (+ 2514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.704365 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.707330 (+ 2965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.707351 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.709943 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.709956 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.710271 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.710340 (+ 69us) client_negotiation.cc:769] Sending connection context
0619 06:32:16.711555 (+ 1215us) client_negotiation.cc:240] Negotiation successful
0619 06:32:16.711784 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2293,"mutex_wait_us":1,"thread_start_us":2153,"threads_started":1}
I20260619 06:32:16.716756 18761 tablet_service.cc:1467] Processing CreateTablet for tablet e7588488cd66454e9132ad05453a00b2 (DEFAULT_TABLE table=test-table [id=71ed6747aeef4eb19b412f46448f9a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:16.718168 18761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7588488cd66454e9132ad05453a00b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:16.718732 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.699347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.0.97.67:38999)
0619 06:32:16.700084 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:16.700092 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:16.710367 (+ 10275us) server_negotiation.cc:407] Connection header received
0619 06:32:16.710909 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:16.710920 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:16.711184 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:16.711349 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:16.712499 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.714109 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.716427 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.717814 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.717951 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.718175 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:16.718356 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 06:32:16.718485 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":414,"threads_started":1}
I20260619 06:32:16.719317 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.698559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38999 (local address 127.0.0.1:39160)
0619 06:32:16.710229 (+ 11670us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:16.710252 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:16.710675 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:16.711642 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:16.711649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:16.711760 (+ 111us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:16.712236 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.712251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.714277 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.714282 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.716270 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.716284 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.717995 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.718001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.718166 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.718201 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:32:16.718943 (+ 742us) client_negotiation.cc:240] Negotiation successful
0619 06:32:16.719065 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11486,"thread_start_us":11843,"threads_started":1}
I20260619 06:32:16.720279 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.700865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43325 (local address 127.0.0.1:46018)
0619 06:32:16.712660 (+ 11795us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:16.712678 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:16.712810 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:16.714302 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:16.714307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:16.714327 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:16.714557 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.714570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.715865 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.715870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.717555 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:16.717569 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.719556 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.719565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:16.719749 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.719779 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 06:32:16.719876 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:32:16.719996 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11678,"thread_start_us":12209,"threads_started":1}
I20260619 06:32:16.720379 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:16.700886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.0.97.66:43325)
0619 06:32:16.713552 (+ 12666us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:16.713559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:16.713591 (+ 32us) server_negotiation.cc:407] Connection header received
0619 06:32:16.713659 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:16.713665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:16.713872 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:16.714047 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:16.714726 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.715726 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.717726 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:16.719198 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:16.719337 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:16.719711 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:16.719931 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 06:32:16.720088 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12513,"thread_start_us":12395,"threads_started":1}
I20260619 06:32:16.722745 19028 tablet_service.cc:1467] Processing CreateTablet for tablet e7588488cd66454e9132ad05453a00b2 (DEFAULT_TABLE table=test-table [id=71ed6747aeef4eb19b412f46448f9a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:16.723778 19028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7588488cd66454e9132ad05453a00b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:16.723676 18895 tablet_service.cc:1467] Processing CreateTablet for tablet e7588488cd66454e9132ad05453a00b2 (DEFAULT_TABLE table=test-table [id=71ed6747aeef4eb19b412f46448f9a14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:16.724879 18895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7588488cd66454e9132ad05453a00b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:16.733917 19111 tablet_bootstrap.cc:492] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Bootstrap starting.
I20260619 06:32:16.736523 19112 tablet_bootstrap.cc:492] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Bootstrap starting.
I20260619 06:32:16.738440 19111 tablet_bootstrap.cc:654] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:16.738485 19113 tablet_bootstrap.cc:492] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Bootstrap starting.
I20260619 06:32:16.739845 19112 tablet_bootstrap.cc:654] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:16.740516 19111 log.cc:824] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:16.740840 19112 log.cc:824] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:16.742005 19113 tablet_bootstrap.cc:654] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:16.742906 19113 log.cc:824] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:16.743513 19111 tablet_bootstrap.cc:492] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: No bootstrap required, opened a new log
I20260619 06:32:16.743722 19112 tablet_bootstrap.cc:492] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: No bootstrap required, opened a new log
I20260619 06:32:16.743759 19111 ts_tablet_manager.cc:1397] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260619 06:32:16.744004 19112 ts_tablet_manager.cc:1397] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 06:32:16.745333 19113 tablet_bootstrap.cc:492] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: No bootstrap required, opened a new log
I20260619 06:32:16.745618 19113 ts_tablet_manager.cc:1397] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260619 06:32:16.752868 19113 raft_consensus.cc:348] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:16.753325 19113 raft_consensus.cc:374] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:16.753428 19113 raft_consensus.cc:729] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fb6ca537454e63b293ff5c41f99cb2, State: Initialized, Role: FOLLOWER
I20260619 06:32:16.754041 19113 consensus_queue.cc:260] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:16.754473 19111 raft_consensus.cc:348] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:16.754473 19112 raft_consensus.cc:348] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:16.754871 19112 raft_consensus.cc:374] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:16.754873 19111 raft_consensus.cc:374] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:16.754976 19111 raft_consensus.cc:729] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ec66c7510440378a1646aaac2d7f0b, State: Initialized, Role: FOLLOWER
I20260619 06:32:16.754969 19112 raft_consensus.cc:729] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87711bf110904224a43e101461ae1d20, State: Initialized, Role: FOLLOWER
I20260619 06:32:16.755194 18960 heartbeater.cc:503] Master 127.0.97.126:45269 was elected leader, sending a full tablet report...
I20260619 06:32:16.755754 19113 ts_tablet_manager.cc:1428] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260619 06:32:16.755582 19112 consensus_queue.cc:260] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:16.755582 19111 consensus_queue.cc:260] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:16.757380 19093 heartbeater.cc:503] Master 127.0.97.126:45269 was elected leader, sending a full tablet report...
I20260619 06:32:16.757524 19112 ts_tablet_manager.cc:1428] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260619 06:32:16.757808 19111 ts_tablet_manager.cc:1428] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260619 06:32:16.885288 19094 tablet.cc:2369] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:32:16.892130 18961 tablet.cc:2369] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:32:16.897941 18827 tablet.cc:2369] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:17.054617 19117 raft_consensus.cc:482] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:32:17.054935 19117 raft_consensus.cc:504] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:17.056582 19117 leader_election.cc:288] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2ec66c7510440378a1646aaac2d7f0b (127.0.97.65:42277), 87711bf110904224a43e101461ae1d20 (127.0.97.67:38999)
I20260619 06:32:17.064283 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.057229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:46375 (local address 127.0.97.65:42277)
0619 06:32:17.057465 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:17.057475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:17.058217 (+ 742us) server_negotiation.cc:407] Connection header received
0619 06:32:17.058386 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:17.058390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:17.058453 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:17.058550 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:17.059224 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.059941 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.062320 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.063348 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.063482 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.063586 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:17.063895 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 06:32:17.064015 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:32:17.064302 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.057273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:51685 (local address 127.0.97.67:38999)
0619 06:32:17.057453 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:17.057461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:17.058103 (+ 642us) server_negotiation.cc:407] Connection header received
0619 06:32:17.058336 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:17.058340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:17.058397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:17.058501 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:17.059212 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.059941 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.062267 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.063077 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.063227 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.063340 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:17.063908 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 06:32:17.064032 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260619 06:32:17.064396 19120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.057154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:42277 (local address 127.0.97.66:46375)
0619 06:32:17.058105 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:17.058119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:17.058233 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:17.058561 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:17.058565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:17.058586 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:17.059073 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:17.059083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.060075 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.060083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:17.062173 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:17.062184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.063561 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.063568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:17.063750 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.063772 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 06:32:17.063870 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 06:32:17.063986 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":88,"thread_start_us":396,"threads_started":1}
I20260619 06:32:17.064953 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.056844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:38999 (local address 127.0.97.66:51685)
0619 06:32:17.057976 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:17.057997 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:17.058202 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:17.058514 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:17.058518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:17.058541 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:17.059067 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:17.059077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.060067 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.060072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:17.062129 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:17.062140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.063483 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.063488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:17.063694 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.063758 (+ 64us) client_negotiation.cc:769] Sending connection context
0619 06:32:17.064449 (+ 691us) client_negotiation.cc:240] Negotiation successful
0619 06:32:17.064591 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":58,"thread_start_us":997,"threads_started":1}
I20260619 06:32:17.065541 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7588488cd66454e9132ad05453a00b2" candidate_uuid: "04fb6ca537454e63b293ff5c41f99cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ec66c7510440378a1646aaac2d7f0b" is_pre_election: true
I20260619 06:32:17.066131 18781 raft_consensus.cc:2454] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fb6ca537454e63b293ff5c41f99cb2 in term 0.
I20260619 06:32:17.066094 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7588488cd66454e9132ad05453a00b2" candidate_uuid: "04fb6ca537454e63b293ff5c41f99cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87711bf110904224a43e101461ae1d20" is_pre_election: true
I20260619 06:32:17.066540 19048 raft_consensus.cc:2454] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04fb6ca537454e63b293ff5c41f99cb2 in term 0.
I20260619 06:32:17.066812 18849 leader_election.cc:302] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fb6ca537454e63b293ff5c41f99cb2, d2ec66c7510440378a1646aaac2d7f0b; no voters:
I20260619 06:32:17.067236 19117 raft_consensus.cc:2781] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:32:17.067384 19117 raft_consensus.cc:482] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:32:17.067514 19117 raft_consensus.cc:3037] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:17.069802 19117 raft_consensus.cc:504] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:17.070415 19117 leader_election.cc:288] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [CANDIDATE]: Term 1 election: Requested vote from peers d2ec66c7510440378a1646aaac2d7f0b (127.0.97.65:42277), 87711bf110904224a43e101461ae1d20 (127.0.97.67:38999)
I20260619 06:32:17.070816 18781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7588488cd66454e9132ad05453a00b2" candidate_uuid: "04fb6ca537454e63b293ff5c41f99cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2ec66c7510440378a1646aaac2d7f0b"
I20260619 06:32:17.071007 18781 raft_consensus.cc:3037] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:17.070976 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7588488cd66454e9132ad05453a00b2" candidate_uuid: "04fb6ca537454e63b293ff5c41f99cb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87711bf110904224a43e101461ae1d20"
I20260619 06:32:17.071192 19048 raft_consensus.cc:3037] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:17.073523 19048 raft_consensus.cc:2454] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fb6ca537454e63b293ff5c41f99cb2 in term 1.
I20260619 06:32:17.073930 18851 leader_election.cc:302] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04fb6ca537454e63b293ff5c41f99cb2, 87711bf110904224a43e101461ae1d20; no voters:
I20260619 06:32:17.074065 18781 raft_consensus.cc:2454] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04fb6ca537454e63b293ff5c41f99cb2 in term 1.
I20260619 06:32:17.074187 19117 raft_consensus.cc:2781] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:17.074764 19117 raft_consensus.cc:686] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 1 LEADER]: Becoming Leader. State: Replica: 04fb6ca537454e63b293ff5c41f99cb2, State: Running, Role: LEADER
I20260619 06:32:17.075289 19117 consensus_queue.cc:237] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } }
I20260619 06:32:17.079947 18636 catalog_manager.cc:5495] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04fb6ca537454e63b293ff5c41f99cb2 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "04fb6ca537454e63b293ff5c41f99cb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04fb6ca537454e63b293ff5c41f99cb2" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 } health_report { overall_health: UNKNOWN } } }
I20260619 06:32:17.162027 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.156628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.0.97.66:43325)
0619 06:32:17.156836 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:17.156843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:17.156918 (+ 75us) server_negotiation.cc:407] Connection header received
0619 06:32:17.157166 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:17.157172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:17.157259 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:17.157398 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:32:17.157770 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.158494 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.159876 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.160281 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.160412 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.160973 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:32:17.161457 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:32:17.161643 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:17.161757 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 06:32:17.161801 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 06:32:17.170145 18781 raft_consensus.cc:1261] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 1 FOLLOWER]: Refusing update from remote peer 04fb6ca537454e63b293ff5c41f99cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:17.170130 19048 raft_consensus.cc:1261] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 1 FOLLOWER]: Refusing update from remote peer 04fb6ca537454e63b293ff5c41f99cb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:17.171190 19122 consensus_queue.cc:1035] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2ec66c7510440378a1646aaac2d7f0b" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 42277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:17.171562 19117 consensus_queue.cc:1035] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87711bf110904224a43e101461ae1d20" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 38999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:17.195250 18637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:17.195519 18637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:17.198782 18637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5e58deceaec42bab7b5bee6fac6f70f: Sending DeleteTablet for 3 replicas of tablet e7588488cd66454e9132ad05453a00b2
I20260619 06:32:17.199994 18761 tablet_service.cc:1514] Processing DeleteTablet for tablet e7588488cd66454e9132ad05453a00b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:33952
I20260619 06:32:17.200243 19028 tablet_service.cc:1514] Processing DeleteTablet for tablet e7588488cd66454e9132ad05453a00b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:39160
I20260619 06:32:17.200690 18895 tablet_service.cc:1514] Processing DeleteTablet for tablet e7588488cd66454e9132ad05453a00b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:17 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:46018
I20260619 06:32:17.200834 19134 tablet_replica.cc:330] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: stopping tablet replica
I20260619 06:32:17.201218 19134 raft_consensus.cc:2229] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:17.201766 19134 raft_consensus.cc:2258] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:17.204203 19134 ts_tablet_manager.cc:1905] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:17.206779 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18698
I20260619 06:32:17.212867 19134 ts_tablet_manager.cc:1918] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:17.213583 19136 tablet_replica.cc:330] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: stopping tablet replica
I20260619 06:32:17.213837 19135 tablet_replica.cc:330] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: stopping tablet replica
I20260619 06:32:17.213976 19136 raft_consensus.cc:2229] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:32:17.214344 19135 raft_consensus.cc:2229] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:17.214610 19136 raft_consensus.cc:2258] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:17.213021 19134 log.cc:1196] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/e7588488cd66454e9132ad05453a00b2
I20260619 06:32:17.214910 19135 raft_consensus.cc:2258] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:17.215433 19134 ts_tablet_manager.cc:1939] T e7588488cd66454e9132ad05453a00b2 P d2ec66c7510440378a1646aaac2d7f0b: Deleting consensus metadata
I20260619 06:32:17.217129 19136 ts_tablet_manager.cc:1905] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:17.217178 18622 catalog_manager.cc:4841] TS d2ec66c7510440378a1646aaac2d7f0b (127.0.97.65:42277): tablet e7588488cd66454e9132ad05453a00b2 (table test-table [id=71ed6747aeef4eb19b412f46448f9a14]) successfully deleted
I20260619 06:32:17.217386 19135 ts_tablet_manager.cc:1905] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:17.224054 19135 ts_tablet_manager.cc:1918] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:17.224289 19135 log.cc:1196] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/e7588488cd66454e9132ad05453a00b2
I20260619 06:32:17.224414 19136 ts_tablet_manager.cc:1918] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:17.224571 19136 log.cc:1196] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/e7588488cd66454e9132ad05453a00b2
I20260619 06:32:17.224958 19135 ts_tablet_manager.cc:1939] T e7588488cd66454e9132ad05453a00b2 P 87711bf110904224a43e101461ae1d20: Deleting consensus metadata
I20260619 06:32:17.225091 19136 ts_tablet_manager.cc:1939] T e7588488cd66454e9132ad05453a00b2 P 04fb6ca537454e63b293ff5c41f99cb2: Deleting consensus metadata
Jun 19 06:32:17 dist-test-slave-4t1f krb5kdc[18592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:32:17.226748 18624 catalog_manager.cc:4841] TS 87711bf110904224a43e101461ae1d20 (127.0.97.67:38999): tablet e7588488cd66454e9132ad05453a00b2 (table test-table [id=71ed6747aeef4eb19b412f46448f9a14]) successfully deleted
I20260619 06:32:17.226753 18622 catalog_manager.cc:4841] TS 04fb6ca537454e63b293ff5c41f99cb2 (127.0.97.66:43325): tablet e7588488cd66454e9132ad05453a00b2 (table test-table [id=71ed6747aeef4eb19b412f46448f9a14]) successfully deleted
I20260619 06:32:17.233203 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.207570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.0.97.65:42277)
0619 06:32:17.207720 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:17.207726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:17.213806 (+ 6080us) server_negotiation.cc:407] Connection header received
0619 06:32:17.214072 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:17.214078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:17.214181 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:17.214256 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:17.218687 (+ 4431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.219777 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.221620 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.222255 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.222348 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.227715 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:17.227732 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:17.227744 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:17.227773 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:17.230526 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:17.231183 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:17.231190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:17.231192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:17.231312 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:17.231680 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:17.231687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:17.231694 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:17.232006 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:17.232323 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:17.232585 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 06:32:17.232788 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:32:17.233861 18821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:17.323592 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18698
I20260619 06:32:17.345856 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18832
Jun 19 06:32:17 dist-test-slave-4t1f krb5kdc[18592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:32:17.363313 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.346828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.97.66:43325)
0619 06:32:17.347038 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:17.347044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:17.347066 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:32:17.347140 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:17.347146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:17.347246 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:17.347361 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:17.348663 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.349984 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.351332 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.351982 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.352110 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.356614 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:17.356629 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:17.356654 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:17.356701 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:17.360632 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:17.361304 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:17.361313 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:17.361317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:17.361457 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:17.361810 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:17.361815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:17.361819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:17.362233 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:17.362558 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:17.362697 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 06:32:17.362917 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:32:17.364066 18955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:17.474979 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18832
I20260619 06:32:17.490439 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18965
Jun 19 06:32:17 dist-test-slave-4t1f krb5kdc[18592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850734, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:32:17.505872 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:17.491160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39176 (local address 127.0.97.67:38999)
0619 06:32:17.491342 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:17.491350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:17.491420 (+ 70us) server_negotiation.cc:407] Connection header received
0619 06:32:17.491628 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:17.491633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:17.491708 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:17.491835 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:17.493199 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.493897 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.495300 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:17.495702 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:17.495838 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:17.500133 (+ 4295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:17.500148 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:17.500163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:17.500194 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:17.502934 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:17.503696 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:17.503704 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:17.503706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:17.503837 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:17.504224 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:17.504230 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:17.504233 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:17.504655 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:17.504947 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:17.505290 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 06:32:17.505505 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:32:17.506556 19088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:17.595294 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18965
I20260619 06:32:17.611454 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 18605
I20260619 06:32:17.612501 18667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:17.722554 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 18605
2026-06-19T06:32:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4403 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T06:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:17Z Disabled control of system clock
I20260619 06:32:17.795454 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33061
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:32:18.112286 19145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:18.112553 19145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:18.112632 19145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:18.119554 19145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:18.119647 19145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:18.119683 19145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:18.119740 19145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:18.119782 19145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:18.128257 19145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:18.129354 19145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:18.130993 19145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:18.142419 19150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:18.142457 19153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:18.142496 19151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:18.143859 19145 server_base.cc:1034] running on GCE node
I20260619 06:32:18.144563 19145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:18.145864 19145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:18.147106 19145 hybrid_clock.cc:648] HybridClock initialized: now 1781850738147074 us; error 45 us; skew 500 ppm
I20260619 06:32:18.147425 19145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:18.149569 19145 webserver.cc:460] Webserver started at http://127.0.97.126:38321/ using document root <none> and password file <none>
I20260619 06:32:18.150194 19145 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:18.150321 19145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:18.150637 19145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:18.153445 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "35561ff43db4438ca247f670f197ce71"
format_stamp: "Formatted at 2026-06-19 06:32:18 on dist-test-slave-4t1f"
server_key: "c07b11bc6b7796b1d534dc05fe970ce4"
server_key_iv: "57a0e07ee455e1120d04090e53089c4b"
server_key_version: "encryptionkey@0"
I20260619 06:32:18.154181 19145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "35561ff43db4438ca247f670f197ce71"
format_stamp: "Formatted at 2026-06-19 06:32:18 on dist-test-slave-4t1f"
server_key: "c07b11bc6b7796b1d534dc05fe970ce4"
server_key_iv: "57a0e07ee455e1120d04090e53089c4b"
server_key_version: "encryptionkey@0"
I20260619 06:32:18.158859 19145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:32:18.161971 19159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:18.163245 19145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:18.163424 19145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "35561ff43db4438ca247f670f197ce71"
format_stamp: "Formatted at 2026-06-19 06:32:18 on dist-test-slave-4t1f"
server_key: "c07b11bc6b7796b1d534dc05fe970ce4"
server_key_iv: "57a0e07ee455e1120d04090e53089c4b"
server_key_version: "encryptionkey@0"
I20260619 06:32:18.163640 19145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:18.184549 19145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:18.185374 19145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:18.185647 19145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:18.203243 19145 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33061
I20260619 06:32:18.203275 19210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33061 every 8 connection(s)
I20260619 06:32:18.204483 19145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:18.204823 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 19145
I20260619 06:32:18.204989 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:18.205544 389 external_mini_cluster.cc:1428] Setting key ea513b96415dbc9bff1ef62fd4bd26ce
I20260619 06:32:18.208868 19211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:18.220726 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:18.207361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41400 (local address 127.0.97.126:33061)
0619 06:32:18.208355 (+ 994us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:18.208370 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:18.208427 (+ 57us) server_negotiation.cc:407] Connection header received
0619 06:32:18.210768 (+ 2341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:18.210801 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:18.211357 (+ 556us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:18.211801 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:18.212317 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:18.214353 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:18.215698 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:18.216709 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:18.216904 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:18.218083 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:18.218093 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:18.218114 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:18.218825 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:18.218922 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:18.218945 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:18.219088 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:18.219222 (+ 134us) server_negotiation.cc:299] Negotiation successful
0619 06:32:18.219537 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":326,"threads_started":1}
I20260619 06:32:18.222389 19211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71: Bootstrap starting.
I20260619 06:32:18.225400 19211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:18.226234 19211 log.cc:824] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:18.228600 19211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71: No bootstrap required, opened a new log
I20260619 06:32:18.235610 19211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35561ff43db4438ca247f670f197ce71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33061 } }
I20260619 06:32:18.235920 19211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:18.235997 19211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35561ff43db4438ca247f670f197ce71, State: Initialized, Role: FOLLOWER
I20260619 06:32:18.236429 19211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35561ff43db4438ca247f670f197ce71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33061 } }
I20260619 06:32:18.236605 19211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:18.236711 19211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:18.236827 19211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:18.238806 19211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35561ff43db4438ca247f670f197ce71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33061 } }
I20260619 06:32:18.239195 19211 leader_election.cc:302] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35561ff43db4438ca247f670f197ce71; no voters:
I20260619 06:32:18.239717 19211 leader_election.cc:288] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:18.239852 19216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:18.240577 19216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [term 1 LEADER]: Becoming Leader. State: Replica: 35561ff43db4438ca247f670f197ce71, State: Running, Role: LEADER
I20260619 06:32:18.241161 19216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35561ff43db4438ca247f670f197ce71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33061 } }
I20260619 06:32:18.241848 19211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:18.244024 19217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35561ff43db4438ca247f670f197ce71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35561ff43db4438ca247f670f197ce71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33061 } } }
I20260619 06:32:18.244207 19218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35561ff43db4438ca247f670f197ce71. Latest consensus state: current_term: 1 leader_uuid: "35561ff43db4438ca247f670f197ce71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35561ff43db4438ca247f670f197ce71" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33061 } } }
I20260619 06:32:18.244668 19218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:18.244779 19217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:18.246088 19226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:18.251101 19226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:18.259493 19226 catalog_manager.cc:1347] Generated new cluster ID: dc1cd5f2d5d942ddafd316872e93a985
I20260619 06:32:18.259593 19226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:18.275627 19226 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:18.277340 19226 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:18.284802 19226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71: Generated new TSK 0
I20260619 06:32:18.285729 19226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:18.293610 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33061
--builtin_ntp_servers=127.0.97.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:32:18.612632 19235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:18.612939 19235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:18.613149 19235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:18.620337 19235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:18.620464 19235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:18.620644 19235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:32:18.629417 19235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:18.630610 19235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:18.632332 19235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:18.644934 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:18.644912 19243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:18.644972 19241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:18.645762 19235 server_base.cc:1034] running on GCE node
I20260619 06:32:18.646385 19235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:18.647243 19235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:18.648483 19235 hybrid_clock.cc:648] HybridClock initialized: now 1781850738648426 us; error 45 us; skew 500 ppm
I20260619 06:32:18.648834 19235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:18.650961 19235 webserver.cc:460] Webserver started at http://127.0.97.65:42981/ using document root <none> and password file <none>
I20260619 06:32:18.651609 19235 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:18.651731 19235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:18.652076 19235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:18.654778 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "fb01960e095047d2b2032f3cb1c78c21"
format_stamp: "Formatted at 2026-06-19 06:32:18 on dist-test-slave-4t1f"
server_key: "1629405ae6105b23b86da9798fd72d33"
server_key_iv: "c58baa85823ca3326dcca0e8057cbbe6"
server_key_version: "encryptionkey@0"
I20260619 06:32:18.655520 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "fb01960e095047d2b2032f3cb1c78c21"
format_stamp: "Formatted at 2026-06-19 06:32:18 on dist-test-slave-4t1f"
server_key: "1629405ae6105b23b86da9798fd72d33"
server_key_iv: "c58baa85823ca3326dcca0e8057cbbe6"
server_key_version: "encryptionkey@0"
I20260619 06:32:18.660336 19235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 06:32:18.664064 19249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:18.665369 19235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 06:32:18.665537 19235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "fb01960e095047d2b2032f3cb1c78c21"
format_stamp: "Formatted at 2026-06-19 06:32:18 on dist-test-slave-4t1f"
server_key: "1629405ae6105b23b86da9798fd72d33"
server_key_iv: "c58baa85823ca3326dcca0e8057cbbe6"
server_key_version: "encryptionkey@0"
I20260619 06:32:18.665741 19235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:18.692761 19235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:18.693553 19235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:18.693799 19235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:18.694754 19235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:18.696233 19235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:18.696364 19235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:18.696498 19235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:18.696585 19235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:18.728950 19235 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:39701
I20260619 06:32:18.729038 19361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:39701 every 8 connection(s)
I20260619 06:32:18.730229 19235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:32:18.738672 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 19235
I20260619 06:32:18.738865 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:32:18.739394 389 external_mini_cluster.cc:1428] Setting key 3c036a70cc3a710992478353a5fd0719
I20260619 06:32:18.742204 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:18.732603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:42813 (local address 127.0.97.126:33061)
0619 06:32:18.733143 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:18.733151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:18.734288 (+ 1137us) server_negotiation.cc:407] Connection header received
0619 06:32:18.735912 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:18.735915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:18.735973 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:18.736063 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:18.736978 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:18.737710 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:18.739272 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:18.739873 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:18.740009 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:18.741426 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:18.741431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:18.741434 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:18.741449 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:18.741493 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:18.741505 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:18.741573 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:18.741845 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 06:32:18.741913 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":273,"threads_started":1}
I20260619 06:32:18.742388 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33061
--builtin_ntp_servers=127.0.97.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:32:18.743630 19365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:18.733262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33061 (local address 127.0.97.65:42813)
0619 06:32:18.734144 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:18.734183 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:18.735697 (+ 1514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:18.736257 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:18.736264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:18.736344 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:18.736843 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:18.736862 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:18.737830 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:18.737833 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:18.739139 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:18.739152 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:18.740103 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:18.740110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:18.740366 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:18.741106 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:18.741110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:18.741166 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:18.741235 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:18.741584 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:18.741595 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:18.741602 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 06:32:18.741821 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 06:32:18.742230 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":272,"threads_started":1}
I20260619 06:32:18.745113 19362 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33061
I20260619 06:32:18.745416 19362 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:18.746126 19362 heartbeater.cc:511] Master 127.0.97.126:33061 requested a full tablet report, sending...
I20260619 06:32:18.747987 19176 ts_manager.cc:194] Registered new tserver with Master: fb01960e095047d2b2032f3cb1c78c21 (127.0.97.65:39701)
I20260619 06:32:18.749550 19176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:42813
W20260619 06:32:19.056731 19366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:19.056989 19366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:19.057142 19366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:19.064193 19366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:19.064285 19366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:19.064487 19366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:32:19.073731 19366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:19.074949 19366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:19.076683 19366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:19.089525 19374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:19.089548 19372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:19.090113 19371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:19.090163 19366 server_base.cc:1034] running on GCE node
I20260619 06:32:19.090791 19366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:19.091641 19366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:19.092832 19366 hybrid_clock.cc:648] HybridClock initialized: now 1781850739092795 us; error 34 us; skew 500 ppm
I20260619 06:32:19.093186 19366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:19.095227 19366 webserver.cc:460] Webserver started at http://127.0.97.66:42919/ using document root <none> and password file <none>
I20260619 06:32:19.095965 19366 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:19.096074 19366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:19.096393 19366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:19.099118 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "c1258194ffbe4243b162bb0511271b36"
format_stamp: "Formatted at 2026-06-19 06:32:19 on dist-test-slave-4t1f"
server_key: "7feb1ebc30354db25b7a85bd7a0df369"
server_key_iv: "d8a2d5e17ea87eb4add46bddf1065ae3"
server_key_version: "encryptionkey@0"
I20260619 06:32:19.099854 19366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "c1258194ffbe4243b162bb0511271b36"
format_stamp: "Formatted at 2026-06-19 06:32:19 on dist-test-slave-4t1f"
server_key: "7feb1ebc30354db25b7a85bd7a0df369"
server_key_iv: "d8a2d5e17ea87eb4add46bddf1065ae3"
server_key_version: "encryptionkey@0"
I20260619 06:32:19.104982 19366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:32:19.107990 19380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:19.109233 19366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:32:19.109395 19366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "c1258194ffbe4243b162bb0511271b36"
format_stamp: "Formatted at 2026-06-19 06:32:19 on dist-test-slave-4t1f"
server_key: "7feb1ebc30354db25b7a85bd7a0df369"
server_key_iv: "d8a2d5e17ea87eb4add46bddf1065ae3"
server_key_version: "encryptionkey@0"
I20260619 06:32:19.109601 19366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:19.124693 19366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:19.125466 19366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:19.125741 19366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:19.126595 19366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:19.127878 19366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:19.128014 19366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:19.128153 19366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:19.128242 19366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:19.159734 19366 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:38275
I20260619 06:32:19.159785 19492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:38275 every 8 connection(s)
I20260619 06:32:19.160939 19366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:32:19.166733 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 19366
I20260619 06:32:19.166916 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:32:19.167347 389 external_mini_cluster.cc:1428] Setting key 55c134961a1f67987150af975027d943
I20260619 06:32:19.170413 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33061
--builtin_ntp_servers=127.0.97.84:40165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:32:19.173883 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.163311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:39437 (local address 127.0.97.126:33061)
0619 06:32:19.163499 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.163503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.164782 (+ 1279us) server_negotiation.cc:407] Connection header received
0619 06:32:19.166561 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.166566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.166693 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.166807 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:19.167960 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.168954 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.170864 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.171633 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.171766 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.172987 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:19.172996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:19.173003 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:19.173027 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:19.173106 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:19.173133 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:19.173221 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.173405 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.173509 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:32:19.174672 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.163824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33061 (local address 127.0.97.66:39437)
0619 06:32:19.164642 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:19.164676 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:19.166314 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:19.167054 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:19.167070 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:19.167158 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:19.167809 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.167826 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.169183 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.169189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.170729 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.170741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.171821 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.171826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.172003 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.172677 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:19.172680 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:19.172714 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:19.172779 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:19.173215 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:19.173222 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:19.173228 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 06:32:19.173362 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 06:32:19.173620 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":249,"threads_started":1}
I20260619 06:32:19.176045 19493 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33061
I20260619 06:32:19.176323 19493 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:19.177028 19493 heartbeater.cc:511] Master 127.0.97.126:33061 requested a full tablet report, sending...
I20260619 06:32:19.178345 19176 ts_manager.cc:194] Registered new tserver with Master: c1258194ffbe4243b162bb0511271b36 (127.0.97.66:38275)
I20260619 06:32:19.179136 19176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:39437
W20260619 06:32:19.487802 19496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:19.488067 19496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:19.488214 19496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:19.495424 19496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:19.495514 19496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:19.495729 19496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:32:19.504930 19496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:19.506206 19496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:19.507881 19496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:19.520820 19501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:19.520907 19504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:19.520875 19502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:19.521763 19496 server_base.cc:1034] running on GCE node
I20260619 06:32:19.522306 19496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:19.523226 19496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:19.524447 19496 hybrid_clock.cc:648] HybridClock initialized: now 1781850739524402 us; error 53 us; skew 500 ppm
I20260619 06:32:19.524741 19496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:19.526859 19496 webserver.cc:460] Webserver started at http://127.0.97.67:39957/ using document root <none> and password file <none>
I20260619 06:32:19.527506 19496 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:19.527637 19496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:19.527945 19496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:19.530638 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "2a64dfc0a8404643a61f06640321df57"
format_stamp: "Formatted at 2026-06-19 06:32:19 on dist-test-slave-4t1f"
server_key: "637b4dc199cbf8d548aaf0adcb4f2610"
server_key_iv: "6914a6fd42b53f2a2f2afe9b741f4233"
server_key_version: "encryptionkey@0"
I20260619 06:32:19.531376 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "2a64dfc0a8404643a61f06640321df57"
format_stamp: "Formatted at 2026-06-19 06:32:19 on dist-test-slave-4t1f"
server_key: "637b4dc199cbf8d548aaf0adcb4f2610"
server_key_iv: "6914a6fd42b53f2a2f2afe9b741f4233"
server_key_version: "encryptionkey@0"
I20260619 06:32:19.536021 19496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 06:32:19.539196 19510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:19.540418 19496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:32:19.540577 19496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "2a64dfc0a8404643a61f06640321df57"
format_stamp: "Formatted at 2026-06-19 06:32:19 on dist-test-slave-4t1f"
server_key: "637b4dc199cbf8d548aaf0adcb4f2610"
server_key_iv: "6914a6fd42b53f2a2f2afe9b741f4233"
server_key_version: "encryptionkey@0"
I20260619 06:32:19.540786 19496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:19.556994 19496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:19.557745 19496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:19.558010 19496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:19.558878 19496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:19.560227 19496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:19.560355 19496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:19.560470 19496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:19.560556 19496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:19.592628 19496 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:41139
I20260619 06:32:19.592710 19622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:41139 every 8 connection(s)
I20260619 06:32:19.594055 19496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:32:19.604007 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 19496
I20260619 06:32:19.604173 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:32:19.604598 389 external_mini_cluster.cc:1428] Setting key 495167ebb3e1d2ff6280da87e1650c3a
I20260619 06:32:19.606902 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.597099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:35249 (local address 127.0.97.126:33061)
0619 06:32:19.597269 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.597273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.598560 (+ 1287us) server_negotiation.cc:407] Connection header received
0619 06:32:19.600306 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.600312 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.600396 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.600518 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:19.601592 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.602332 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.603842 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.604409 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.604545 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.606054 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:19.606061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:19.606066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:19.606091 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:19.606123 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:19.606136 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:19.606227 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.606449 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.606556 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:32:19.607820 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.597516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:33061 (local address 127.0.97.67:35249)
0619 06:32:19.598416 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:19.598451 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:19.600039 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:19.600738 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:19.600746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:19.600827 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:19.601423 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.601439 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.602488 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.602492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.603703 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.603714 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.604617 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.604624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.604892 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.605699 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:19.605703 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:19.605752 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:19.605860 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:19.606236 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:19.606247 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:19.606256 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 06:32:19.606418 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:32:19.606774 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":273,"threads_started":1}
I20260619 06:32:19.608978 19623 heartbeater.cc:347] Connected to a master server at 127.0.97.126:33061
I20260619 06:32:19.609279 19623 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:19.609942 19623 heartbeater.cc:511] Master 127.0.97.126:33061 requested a full tablet report, sending...
I20260619 06:32:19.611178 19176 ts_manager.cc:194] Registered new tserver with Master: 2a64dfc0a8404643a61f06640321df57 (127.0.97.67:41139)
I20260619 06:32:19.611943 19176 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:35249
I20260619 06:32:19.619218 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:32:19.628659 19364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.622866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55188 (local address 127.0.97.126:33061)
0619 06:32:19.623052 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.623059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.623448 (+ 389us) server_negotiation.cc:407] Connection header received
0619 06:32:19.623629 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.623634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.623712 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.623818 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:19.624264 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.624924 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.626314 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.626702 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.626833 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.627945 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:19.627949 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:19.627953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:19.627967 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:19.627996 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:19.628007 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:19.628176 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.628350 (+ 174us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.628402 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:32:19.631973 19176 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55188:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:32:19.638429 19176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:32:19.662289 19635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.649764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39701 (local address 127.0.0.1:32904)
0619 06:32:19.650962 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:19.651008 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:19.651239 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:19.654848 (+ 3609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:19.654854 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:19.654876 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:19.655327 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.655341 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.656794 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.656802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.659439 (+ 2637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.659453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.661307 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.661313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.661523 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.661573 (+ 50us) client_negotiation.cc:769] Sending connection context
0619 06:32:19.661706 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 06:32:19.661895 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":129,"thread_start_us":1073,"threads_started":1}
I20260619 06:32:19.662707 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.650978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.0.97.66:38275)
0619 06:32:19.653879 (+ 2901us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.653899 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.653937 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:32:19.654060 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.654070 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.654356 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.655330 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:19.655576 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.656990 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.659997 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.661416 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.661547 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.661808 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.662046 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.662239 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2734,"thread_start_us":2633,"threads_started":1}
I20260619 06:32:19.662763 19638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.650205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:38275 (local address 127.0.0.1:50596)
0619 06:32:19.653621 (+ 3416us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:19.653642 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:19.653777 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:19.654587 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:19.654595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:19.654626 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:19.655058 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.655076 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.657163 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.657173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.659820 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.659833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.661645 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.661652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.661847 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.661881 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:32:19.661996 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 06:32:19.662116 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3274,"mutex_wait_us":276,"thread_start_us":4105,"threads_started":1}
I20260619 06:32:19.662878 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.651368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.0.97.67:41139)
0619 06:32:19.652318 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.652329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.654622 (+ 2293us) server_negotiation.cc:407] Connection header received
0619 06:32:19.654893 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.654902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.655102 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.655278 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:19.656092 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.657581 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.659755 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.660727 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.660889 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.661184 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.662378 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.662537 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":548,"threads_started":1}
I20260619 06:32:19.663048 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.649986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.0.97.65:39701)
0619 06:32:19.654269 (+ 4283us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.654279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.654309 (+ 30us) server_negotiation.cc:407] Connection header received
0619 06:32:19.654405 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.654415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.654647 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.655232 (+ 585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:19.655469 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.656658 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.659657 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.661066 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.661277 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.661502 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.662552 (+ 1050us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.662733 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4116,"thread_start_us":4030,"threads_started":1}
I20260619 06:32:19.663815 19637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.650661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:41139 (local address 127.0.0.1:41758)
0619 06:32:19.654525 (+ 3864us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:19.654540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:19.654739 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:19.655514 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:19.655519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:19.655543 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:19.655897 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.655909 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.657748 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.657753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.659528 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.659540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.661986 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.661992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.662167 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.662198 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:32:19.663381 (+ 1183us) client_negotiation.cc:240] Negotiation successful
0619 06:32:19.663505 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3765,"mutex_wait_us":96,"thread_start_us":2645,"threads_started":1}
I20260619 06:32:19.666939 19428 tablet_service.cc:1467] Processing CreateTablet for tablet 8a78c91cd29b44f298aafd6fe71a24e3 (DEFAULT_TABLE table=test-table [id=b602fb6a7a6d4df7b75fbb9a40cb11c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:19.666942 19297 tablet_service.cc:1467] Processing CreateTablet for tablet 8a78c91cd29b44f298aafd6fe71a24e3 (DEFAULT_TABLE table=test-table [id=b602fb6a7a6d4df7b75fbb9a40cb11c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:19.668427 19297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a78c91cd29b44f298aafd6fe71a24e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:19.668427 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a78c91cd29b44f298aafd6fe71a24e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:19.668257 19558 tablet_service.cc:1467] Processing CreateTablet for tablet 8a78c91cd29b44f298aafd6fe71a24e3 (DEFAULT_TABLE table=test-table [id=b602fb6a7a6d4df7b75fbb9a40cb11c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:19.669500 19558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a78c91cd29b44f298aafd6fe71a24e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:19.680102 19641 tablet_bootstrap.cc:492] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Bootstrap starting.
I20260619 06:32:19.683713 19642 tablet_bootstrap.cc:492] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Bootstrap starting.
I20260619 06:32:19.684551 19641 tablet_bootstrap.cc:654] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:19.685886 19641 log.cc:824] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:19.687055 19642 tablet_bootstrap.cc:654] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:19.687338 19643 tablet_bootstrap.cc:492] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Bootstrap starting.
I20260619 06:32:19.688279 19642 log.cc:824] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:19.689867 19641 tablet_bootstrap.cc:492] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: No bootstrap required, opened a new log
I20260619 06:32:19.690184 19641 ts_tablet_manager.cc:1397] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20260619 06:32:19.690637 19643 tablet_bootstrap.cc:654] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:19.691462 19642 tablet_bootstrap.cc:492] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: No bootstrap required, opened a new log
I20260619 06:32:19.691731 19642 ts_tablet_manager.cc:1397] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260619 06:32:19.692025 19643 log.cc:824] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:19.698580 19643 tablet_bootstrap.cc:492] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: No bootstrap required, opened a new log
I20260619 06:32:19.698858 19643 ts_tablet_manager.cc:1397] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260619 06:32:19.698621 19641 raft_consensus.cc:348] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.699152 19641 raft_consensus.cc:374] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:19.699285 19641 raft_consensus.cc:729] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a64dfc0a8404643a61f06640321df57, State: Initialized, Role: FOLLOWER
I20260619 06:32:19.699913 19641 consensus_queue.cc:260] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.701097 19623 heartbeater.cc:503] Master 127.0.97.126:33061 was elected leader, sending a full tablet report...
I20260619 06:32:19.701767 19641 ts_tablet_manager.cc:1428] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20260619 06:32:19.702939 19642 raft_consensus.cc:348] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.703346 19642 raft_consensus.cc:374] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:19.703449 19642 raft_consensus.cc:729] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1258194ffbe4243b162bb0511271b36, State: Initialized, Role: FOLLOWER
I20260619 06:32:19.704103 19642 consensus_queue.cc:260] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.705785 19493 heartbeater.cc:503] Master 127.0.97.126:33061 was elected leader, sending a full tablet report...
I20260619 06:32:19.706394 19642 ts_tablet_manager.cc:1428] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Time spent starting tablet: real 0.015s user 0.010s sys 0.006s
I20260619 06:32:19.709630 19643 raft_consensus.cc:348] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.710117 19643 raft_consensus.cc:374] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:19.710228 19643 raft_consensus.cc:729] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb01960e095047d2b2032f3cb1c78c21, State: Initialized, Role: FOLLOWER
I20260619 06:32:19.710762 19643 consensus_queue.cc:260] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.711733 19362 heartbeater.cc:503] Master 127.0.97.126:33061 was elected leader, sending a full tablet report...
I20260619 06:32:19.712195 19643 ts_tablet_manager.cc:1428] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260619 06:32:19.732357 19363 tablet.cc:2369] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:32:19.846153 19624 tablet.cc:2369] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:19.907642 19648 raft_consensus.cc:482] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:32:19.907975 19648 raft_consensus.cc:504] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.911486 19648 leader_election.cc:288] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb01960e095047d2b2032f3cb1c78c21 (127.0.97.65:39701), 2a64dfc0a8404643a61f06640321df57 (127.0.97.67:41139)
W20260619 06:32:19.912875 19494 tablet.cc:2369] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:19.921790 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.911226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:36457 (local address 127.0.97.65:39701)
0619 06:32:19.911478 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.911487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.911791 (+ 304us) server_negotiation.cc:407] Connection header received
0619 06:32:19.911893 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.911902 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.912018 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.912140 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:19.914132 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.915535 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.918959 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.920467 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.920625 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.920748 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.921278 (+ 530us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.921446 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:32:19.922220 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.910731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:39701 (local address 127.0.97.66:36457)
0619 06:32:19.911550 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:19.911575 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:19.911758 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:19.912166 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:19.912173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:19.912208 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:19.913956 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.913975 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.915751 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.915760 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.918693 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.918727 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.920715 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.920724 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.921018 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.921125 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 06:32:19.921238 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 06:32:19.921508 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"threads_started":1}
I20260619 06:32:19.923563 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78c91cd29b44f298aafd6fe71a24e3" candidate_uuid: "c1258194ffbe4243b162bb0511271b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb01960e095047d2b2032f3cb1c78c21" is_pre_election: true
I20260619 06:32:19.924136 19317 raft_consensus.cc:2454] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1258194ffbe4243b162bb0511271b36 in term 0.
I20260619 06:32:19.924906 19382 leader_election.cc:302] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1258194ffbe4243b162bb0511271b36, fb01960e095047d2b2032f3cb1c78c21; no voters:
I20260619 06:32:19.925450 19648 raft_consensus.cc:2781] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:32:19.925613 19648 raft_consensus.cc:482] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:32:19.925736 19648 raft_consensus.cc:3037] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:19.927969 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.911681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:54839 (local address 127.0.97.67:41139)
0619 06:32:19.911904 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.911912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.922534 (+ 10622us) server_negotiation.cc:407] Connection header received
0619 06:32:19.922800 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.922808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.922903 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.923015 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:19.923334 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.924405 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.926336 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.927095 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.927233 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.927344 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.927611 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.927740 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:32:19.927990 19651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.910817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:41139 (local address 127.0.97.66:54839)
0619 06:32:19.922450 (+ 11633us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:19.922468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:19.922639 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:19.923044 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:19.923048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:19.923072 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:19.923203 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.923209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.924549 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.924557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.926194 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:19.926206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.927242 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.927248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:19.927460 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.927481 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 06:32:19.927586 (+ 105us) client_negotiation.cc:240] Negotiation successful
0619 06:32:19.927705 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11505,"thread_start_us":364,"threads_started":1}
I20260619 06:32:19.928884 19578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78c91cd29b44f298aafd6fe71a24e3" candidate_uuid: "c1258194ffbe4243b162bb0511271b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a64dfc0a8404643a61f06640321df57" is_pre_election: true
I20260619 06:32:19.929378 19578 raft_consensus.cc:2454] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1258194ffbe4243b162bb0511271b36 in term 0.
I20260619 06:32:19.929651 19648 raft_consensus.cc:504] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.930255 19648 leader_election.cc:288] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [CANDIDATE]: Term 1 election: Requested vote from peers fb01960e095047d2b2032f3cb1c78c21 (127.0.97.65:39701), 2a64dfc0a8404643a61f06640321df57 (127.0.97.67:41139)
I20260619 06:32:19.930596 19317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78c91cd29b44f298aafd6fe71a24e3" candidate_uuid: "c1258194ffbe4243b162bb0511271b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb01960e095047d2b2032f3cb1c78c21"
I20260619 06:32:19.930678 19578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a78c91cd29b44f298aafd6fe71a24e3" candidate_uuid: "c1258194ffbe4243b162bb0511271b36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a64dfc0a8404643a61f06640321df57"
I20260619 06:32:19.930786 19317 raft_consensus.cc:3037] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:19.930887 19578 raft_consensus.cc:3037] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:19.932936 19317 raft_consensus.cc:2454] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1258194ffbe4243b162bb0511271b36 in term 1.
I20260619 06:32:19.933050 19578 raft_consensus.cc:2454] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1258194ffbe4243b162bb0511271b36 in term 1.
I20260619 06:32:19.933343 19382 leader_election.cc:302] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c1258194ffbe4243b162bb0511271b36, fb01960e095047d2b2032f3cb1c78c21; no voters:
I20260619 06:32:19.933624 19648 raft_consensus.cc:2781] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:19.934010 19648 raft_consensus.cc:686] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 1 LEADER]: Becoming Leader. State: Replica: c1258194ffbe4243b162bb0511271b36, State: Running, Role: LEADER
I20260619 06:32:19.934455 19648 consensus_queue.cc:237] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } }
I20260619 06:32:19.939021 19175 catalog_manager.cc:5495] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 reported cstate change: term changed from 0 to 1, leader changed from <none> to c1258194ffbe4243b162bb0511271b36 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "c1258194ffbe4243b162bb0511271b36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1258194ffbe4243b162bb0511271b36" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 38275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 } health_report { overall_health: UNKNOWN } } }
I20260619 06:32:19.957650 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.952363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.0.97.66:38275)
0619 06:32:19.952542 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.952549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:19.952566 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:32:19.952769 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:19.952773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:19.952843 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:19.952983 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:32:19.953423 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.954103 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.955472 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:19.955897 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:19.956030 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:19.956599 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:32:19.957045 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:32:19.957289 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:19.957397 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 06:32:19.957436 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 06:32:19.965049 19317 raft_consensus.cc:1261] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 1 FOLLOWER]: Refusing update from remote peer c1258194ffbe4243b162bb0511271b36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:19.965063 19578 raft_consensus.cc:1261] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 1 FOLLOWER]: Refusing update from remote peer c1258194ffbe4243b162bb0511271b36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:19.966023 19648 consensus_queue.cc:1035] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb01960e095047d2b2032f3cb1c78c21" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 39701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:19.966308 19652 consensus_queue.cc:1035] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a64dfc0a8404643a61f06640321df57" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:19.992152 19175 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:19.992434 19175 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55188:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:19.995946 19175 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 35561ff43db4438ca247f670f197ce71: Sending DeleteTablet for 3 replicas of tablet 8a78c91cd29b44f298aafd6fe71a24e3
I20260619 06:32:19.997049 19297 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a78c91cd29b44f298aafd6fe71a24e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:19 UTC) from {username='slave'} at 127.0.0.1:32904
I20260619 06:32:19.997345 19558 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a78c91cd29b44f298aafd6fe71a24e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:19 UTC) from {username='slave'} at 127.0.0.1:41758
I20260619 06:32:19.997824 19428 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a78c91cd29b44f298aafd6fe71a24e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:19 UTC) from {username='slave'} at 127.0.0.1:50596
I20260619 06:32:19.998382 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 19235
I20260619 06:32:20.005713 19665 tablet_replica.cc:330] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: stopping tablet replica
I20260619 06:32:20.006317 19665 raft_consensus.cc:2229] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:20.006961 19665 raft_consensus.cc:2258] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:20.009285 19666 tablet_replica.cc:330] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: stopping tablet replica
I20260619 06:32:20.009723 19666 raft_consensus.cc:2229] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:32:20.009778 19665 ts_tablet_manager.cc:1905] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:20.010418 19666 raft_consensus.cc:2258] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:20.014659 19666 ts_tablet_manager.cc:1905] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:20.017704 19664 tablet_replica.cc:330] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: stopping tablet replica
I20260619 06:32:20.018282 19664 raft_consensus.cc:2229] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:20.018518 19665 ts_tablet_manager.cc:1918] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:20.018693 19665 log.cc:1196] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/8a78c91cd29b44f298aafd6fe71a24e3
I20260619 06:32:20.018910 19664 raft_consensus.cc:2258] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:20.021350 19664 ts_tablet_manager.cc:1905] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:20.022068 19666 ts_tablet_manager.cc:1918] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:20.022246 19666 log.cc:1196] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/8a78c91cd29b44f298aafd6fe71a24e3
I20260619 06:32:20.022779 19666 ts_tablet_manager.cc:1939] T 8a78c91cd29b44f298aafd6fe71a24e3 P c1258194ffbe4243b162bb0511271b36: Deleting consensus metadata
I20260619 06:32:20.024559 19163 catalog_manager.cc:4841] TS c1258194ffbe4243b162bb0511271b36 (127.0.97.66:38275): tablet 8a78c91cd29b44f298aafd6fe71a24e3 (table test-table [id=b602fb6a7a6d4df7b75fbb9a40cb11c9]) successfully deleted
I20260619 06:32:20.025837 19665 ts_tablet_manager.cc:1939] T 8a78c91cd29b44f298aafd6fe71a24e3 P 2a64dfc0a8404643a61f06640321df57: Deleting consensus metadata
I20260619 06:32:20.027721 19160 catalog_manager.cc:4841] TS 2a64dfc0a8404643a61f06640321df57 (127.0.97.67:41139): tablet 8a78c91cd29b44f298aafd6fe71a24e3 (table test-table [id=b602fb6a7a6d4df7b75fbb9a40cb11c9]) successfully deleted
I20260619 06:32:20.028234 19664 ts_tablet_manager.cc:1918] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:20.028393 19664 log.cc:1196] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/8a78c91cd29b44f298aafd6fe71a24e3
I20260619 06:32:20.028911 19664 ts_tablet_manager.cc:1939] T 8a78c91cd29b44f298aafd6fe71a24e3 P fb01960e095047d2b2032f3cb1c78c21: Deleting consensus metadata
I20260619 06:32:20.030473 19161 catalog_manager.cc:4841] TS fb01960e095047d2b2032f3cb1c78c21 (127.0.97.65:39701): tablet 8a78c91cd29b44f298aafd6fe71a24e3 (table test-table [id=b602fb6a7a6d4df7b75fbb9a40cb11c9]) successfully deleted
I20260619 06:32:20.033232 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:19.999787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.0.97.65:39701)
0619 06:32:19.999976 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:19.999983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:20.025568 (+ 25585us) server_negotiation.cc:407] Connection header received
0619 06:32:20.025754 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:20.025760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:20.025867 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:20.025978 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:20.026704 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:20.027863 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:20.029784 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:20.030439 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:20.030570 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:20.031676 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:20.031686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:20.031703 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:20.032449 (+ 746us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:20.032507 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:20.032531 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:20.032658 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:20.032736 (+ 78us) server_negotiation.cc:299] Negotiation successful
0619 06:32:20.032838 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:32:20.033859 19357 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:20.128556 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 19235
I20260619 06:32:20.152833 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 19366
I20260619 06:32:20.160437 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:20.153648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.0.97.66:38275)
0619 06:32:20.153863 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:20.153869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:20.153888 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:32:20.154007 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:20.154012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:20.154102 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:20.154219 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:20.154662 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:20.155473 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:20.156756 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:20.157255 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:20.157388 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:20.158732 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:20.158741 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:20.158774 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:20.159525 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:20.159593 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:20.159616 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:20.159741 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:20.159895 (+ 154us) server_negotiation.cc:299] Negotiation successful
0619 06:32:20.160005 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260619 06:32:20.161288 19488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:20.271293 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 19366
I20260619 06:32:20.289783 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 19496
I20260619 06:32:20.296546 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:20.290686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.0.97.67:41139)
0619 06:32:20.290899 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:20.290907 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:20.290929 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:32:20.290991 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:20.290996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:20.291117 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:20.291230 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:20.291641 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:20.292341 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:20.293776 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:20.294171 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:20.294301 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:20.295259 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:20.295268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:20.295281 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:20.295827 (+ 546us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:20.295870 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:20.295888 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:20.296004 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:20.296129 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 06:32:20.296245 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:32:20.297269 19618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:20.385015 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 19496
I20260619 06:32:20.401139 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 19145
I20260619 06:32:20.402128 19206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:20.509795 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 19145
2026-06-19T06:32:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2786 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:20 dist-test-slave-4t1f krb5kdc[19674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:20 dist-test-slave-4t1f krb5kdc[19674](info): set up 2 sockets
Jun 19 06:32:20 dist-test-slave-4t1f krb5kdc[19674](info): commencing operation
krb5kdc: starting...
W20260619 06:32:21.617256 389 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:21 dist-test-slave-4t1f krb5kdc[19674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:21.842394 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43949
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:43949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:22.170500 19687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:22.170822 19687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:22.170974 19687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:22.178151 19687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:22.178297 19687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:22.178400 19687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:22.178499 19687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:22.178586 19687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:22.188007 19687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:43949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:43949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:22.189213 19687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:22.190479 19687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:22.201815 19695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:22.201926 19693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:22.202296 19692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:22.202538 19687 server_base.cc:1034] running on GCE node
I20260619 06:32:22.203392 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:22.204672 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:22.205914 19687 hybrid_clock.cc:648] HybridClock initialized: now 1781850742205859 us; error 41 us; skew 500 ppm
I20260619 06:32:22.206231 19687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:22 dist-test-slave-4t1f krb5kdc[19674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:22.209316 19687 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:22.210647 19687 webserver.cc:460] Webserver started at http://127.0.97.126:34571/ using document root <none> and password file <none>
I20260619 06:32:22.211288 19687 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:22.211426 19687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:22.211730 19687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:22.214519 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "c47025979bdc45968557ec68f9584f96"
format_stamp: "Formatted at 2026-06-19 06:32:22 on dist-test-slave-4t1f"
server_key: "671243ac32b553fcbde1ac5d3a81fef4"
server_key_iv: "8238c1eb70d305d8580b9abc612cf92b"
server_key_version: "encryptionkey@0"
I20260619 06:32:22.215247 19687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "c47025979bdc45968557ec68f9584f96"
format_stamp: "Formatted at 2026-06-19 06:32:22 on dist-test-slave-4t1f"
server_key: "671243ac32b553fcbde1ac5d3a81fef4"
server_key_iv: "8238c1eb70d305d8580b9abc612cf92b"
server_key_version: "encryptionkey@0"
I20260619 06:32:22.220046 19687 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:32:22.223227 19702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:22.224462 19687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:32:22.224653 19687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "c47025979bdc45968557ec68f9584f96"
format_stamp: "Formatted at 2026-06-19 06:32:22 on dist-test-slave-4t1f"
server_key: "671243ac32b553fcbde1ac5d3a81fef4"
server_key_iv: "8238c1eb70d305d8580b9abc612cf92b"
server_key_version: "encryptionkey@0"
I20260619 06:32:22.224880 19687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:22.238283 19687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:22.242398 19687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:22.242672 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:22.260548 19687 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:43949
I20260619 06:32:22.260577 19753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:43949 every 8 connection(s)
I20260619 06:32:22.261857 19687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:22.265266 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 19687
I20260619 06:32:22.265339 19754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:22.265463 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:22.265888 389 external_mini_cluster.cc:1428] Setting key 4d386986189f79d697cb867710abd4de
I20260619 06:32:22.279237 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96: Bootstrap starting.
Jun 19 06:32:22 dist-test-slave-4t1f krb5kdc[19674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:22.282894 19754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:22.284250 19754 log.cc:824] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:22.287626 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96: No bootstrap required, opened a new log
I20260619 06:32:22.289649 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:22.267665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.0.97.126:43949)
0619 06:32:22.268515 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:22.268534 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:22.268581 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:32:22.270316 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:22.270342 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:22.270836 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:22.271249 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:22.272599 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:22.274565 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:22.275837 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:22.276917 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:22.277127 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:22.281266 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:22.281276 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:22.281292 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:22.281332 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:22.285247 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:22.286001 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:22.286009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:22.286016 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:22.286211 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:22.286563 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:22.286574 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:22.286577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:22.287257 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:22.287521 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:22.287877 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 06:32:22.288314 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":369,"threads_started":1}
I20260619 06:32:22.296017 19754 raft_consensus.cc:348] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47025979bdc45968557ec68f9584f96" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43949 } }
I20260619 06:32:22.296342 19754 raft_consensus.cc:374] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:22.296412 19754 raft_consensus.cc:729] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c47025979bdc45968557ec68f9584f96, State: Initialized, Role: FOLLOWER
I20260619 06:32:22.296854 19754 consensus_queue.cc:260] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47025979bdc45968557ec68f9584f96" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43949 } }
I20260619 06:32:22.297046 19754 raft_consensus.cc:388] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:22.297224 19754 raft_consensus.cc:482] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:22.297348 19754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:22.299410 19754 raft_consensus.cc:504] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47025979bdc45968557ec68f9584f96" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43949 } }
I20260619 06:32:22.299821 19754 leader_election.cc:302] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c47025979bdc45968557ec68f9584f96; no voters:
I20260619 06:32:22.300516 19754 leader_election.cc:288] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:22.300631 19759 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:22.301342 19759 raft_consensus.cc:686] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [term 1 LEADER]: Becoming Leader. State: Replica: c47025979bdc45968557ec68f9584f96, State: Running, Role: LEADER
I20260619 06:32:22.301877 19759 consensus_queue.cc:237] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47025979bdc45968557ec68f9584f96" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43949 } }
I20260619 06:32:22.302608 19754 sys_catalog.cc:564] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:22.304575 19760 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c47025979bdc45968557ec68f9584f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47025979bdc45968557ec68f9584f96" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43949 } } }
I20260619 06:32:22.305136 19760 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:22.304553 19761 sys_catalog.cc:455] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c47025979bdc45968557ec68f9584f96. Latest consensus state: current_term: 1 leader_uuid: "c47025979bdc45968557ec68f9584f96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c47025979bdc45968557ec68f9584f96" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 43949 } } }
I20260619 06:32:22.305632 19761 sys_catalog.cc:458] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:22.307544 19766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:22.312539 19766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:22.320680 19766 catalog_manager.cc:1347] Generated new cluster ID: c49de1740fdc4e9485290d3f9ed4a0fc
I20260619 06:32:22.320788 19766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:22.341476 19766 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:22.343317 19766 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:22.351786 19766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96: Generated new TSK 0
I20260619 06:32:22.352723 19766 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
Entry for principal kudu/127.0.97.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab.
I20260619 06:32:22.416275 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43949
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:22.741181 19780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:22.741530 19780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:22.741720 19780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:22.748623 19780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:22.748777 19780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:22.748966 19780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:32:22.758380 19780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.65.keytab
--principal=kudu/127.0.97.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:22.759626 19780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:22.760882 19780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:22.773574 19788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:22.773679 19786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:22.773545 19785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:22.774701 19780 server_base.cc:1034] running on GCE node
I20260619 06:32:22.775282 19780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:22.776150 19780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:22.777333 19780 hybrid_clock.cc:648] HybridClock initialized: now 1781850742777283 us; error 50 us; skew 500 ppm
I20260619 06:32:22.777624 19780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:22 dist-test-slave-4t1f krb5kdc[19674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:22.780572 19780 init.cc:377] Logged in from keytab as kudu/127.0.97.65@KRBTEST.COM (short username kudu)
I20260619 06:32:22.781899 19780 webserver.cc:460] Webserver started at http://127.0.97.65:41753/ using document root <none> and password file <none>
I20260619 06:32:22.782534 19780 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:22.782678 19780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:22.782965 19780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:22.785745 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "e08645644ef14f86b74b87991e592915"
format_stamp: "Formatted at 2026-06-19 06:32:22 on dist-test-slave-4t1f"
server_key: "7e1b64d9be71fda525da96be054b90ef"
server_key_iv: "af8cc0f690de8c05e0f46cfd000b44b3"
server_key_version: "encryptionkey@0"
I20260619 06:32:22.786502 19780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "e08645644ef14f86b74b87991e592915"
format_stamp: "Formatted at 2026-06-19 06:32:22 on dist-test-slave-4t1f"
server_key: "7e1b64d9be71fda525da96be054b90ef"
server_key_iv: "af8cc0f690de8c05e0f46cfd000b44b3"
server_key_version: "encryptionkey@0"
I20260619 06:32:22.791404 19780 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:32:22.794694 19795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:22.795944 19780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:32:22.796130 19780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "e08645644ef14f86b74b87991e592915"
format_stamp: "Formatted at 2026-06-19 06:32:22 on dist-test-slave-4t1f"
server_key: "7e1b64d9be71fda525da96be054b90ef"
server_key_iv: "af8cc0f690de8c05e0f46cfd000b44b3"
server_key_version: "encryptionkey@0"
I20260619 06:32:22.796335 19780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:22.810775 19780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:22.814607 19780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:22.814878 19780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:22.815757 19780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:22.816982 19780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:22.817121 19780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:22.817272 19780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:22.817350 19780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:22.849807 19780 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:36795
I20260619 06:32:22.849895 19907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:36795 every 8 connection(s)
I20260619 06:32:22.851046 19780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:32:22.860255 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 19780
I20260619 06:32:22.860427 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:32:22.860980 389 external_mini_cluster.cc:1428] Setting key 54314ef3945bd78f0ff0bc942f61bac5
Jun 19 06:32:22 dist-test-slave-4t1f krb5kdc[19674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.65@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:22.870501 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:22.853567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:40193 (local address 127.0.97.126:43949)
0619 06:32:22.854005 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:22.854010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:22.854916 (+ 906us) server_negotiation.cc:407] Connection header received
0619 06:32:22.856703 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:22.856708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:22.856806 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:22.856931 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:22.858419 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:22.859148 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:22.861162 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:22.861616 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:22.861763 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:22.865128 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:22.865133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:22.865137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:22.865170 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:22.867550 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:22.868253 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:22.868265 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:22.868267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:22.868380 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:22.868795 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:22.868801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:22.868805 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:22.869142 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:22.869292 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:22.869930 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 06:32:22.870154 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":240,"threads_started":1}
I20260619 06:32:22.871381 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:22.854028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43949 (local address 127.0.97.65:40193)
0619 06:32:22.854761 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:22.854798 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:22.856439 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:22.857185 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:22.857194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:22.857692 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:22.858229 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:22.858265 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:22.859291 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:22.859303 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:22.860950 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:22.860966 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:22.861865 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:22.861872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:22.862120 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:22.862826 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:22.862856 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:22.864891 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:22.867707 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:22.867715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:22.867729 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:22.868128 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:22.868490 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:22.868496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:22.868499 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:22.868684 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:22.869275 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:22.869282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:22.869584 (+ 302us) client_negotiation.cc:769] Sending connection context
0619 06:32:22.869836 (+ 252us) client_negotiation.cc:240] Negotiation successful
0619 06:32:22.870326 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":288,"threads_started":1}
I20260619 06:32:22.872844 19908 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43949
I20260619 06:32:22.873180 19908 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:22.873994 19908 heartbeater.cc:511] Master 127.0.97.126:43949 requested a full tablet report, sending...
I20260619 06:32:22.876214 19719 ts_manager.cc:194] Registered new tserver with Master: e08645644ef14f86b74b87991e592915 (127.0.97.65:36795)
I20260619 06:32:22.878424 19719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.65@KRBTEST.COM'} at 127.0.97.65:40193
WARNING: no policy specified for kudu/127.0.97.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
Entry for principal kudu/127.0.97.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab.
I20260619 06:32:22.913522 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43949
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:23.228310 19914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:23.228578 19914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:23.228668 19914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:23.236076 19914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:23.236196 19914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:23.236341 19914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:32:23.245893 19914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.66.keytab
--principal=kudu/127.0.97.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:23.247200 19914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:23.248417 19914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:23.261629 19919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:23.262284 19920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:23.263319 19914 server_base.cc:1034] running on GCE node
W20260619 06:32:23.265612 19922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:23.266342 19914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:23.267423 19914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:23.268663 19914 hybrid_clock.cc:648] HybridClock initialized: now 1781850743268600 us; error 53 us; skew 500 ppm
I20260619 06:32:23.268996 19914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:23 dist-test-slave-4t1f krb5kdc[19674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:23.272562 19914 init.cc:377] Logged in from keytab as kudu/127.0.97.66@KRBTEST.COM (short username kudu)
I20260619 06:32:23.274062 19914 webserver.cc:460] Webserver started at http://127.0.97.66:46111/ using document root <none> and password file <none>
I20260619 06:32:23.274726 19914 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:23.274849 19914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:23.275161 19914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:23.277987 19914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "f586449d78c542aa8baae0b4f0988ea1"
format_stamp: "Formatted at 2026-06-19 06:32:23 on dist-test-slave-4t1f"
server_key: "f42738439d8abbcca31e2f0f4a55f37c"
server_key_iv: "6fdef73cea6edd9148bd44107e3a3b28"
server_key_version: "encryptionkey@0"
I20260619 06:32:23.278784 19914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "f586449d78c542aa8baae0b4f0988ea1"
format_stamp: "Formatted at 2026-06-19 06:32:23 on dist-test-slave-4t1f"
server_key: "f42738439d8abbcca31e2f0f4a55f37c"
server_key_iv: "6fdef73cea6edd9148bd44107e3a3b28"
server_key_version: "encryptionkey@0"
I20260619 06:32:23.283658 19914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 06:32:23.287005 19929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:23.288259 19914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:32:23.288421 19914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "f586449d78c542aa8baae0b4f0988ea1"
format_stamp: "Formatted at 2026-06-19 06:32:23 on dist-test-slave-4t1f"
server_key: "f42738439d8abbcca31e2f0f4a55f37c"
server_key_iv: "6fdef73cea6edd9148bd44107e3a3b28"
server_key_version: "encryptionkey@0"
I20260619 06:32:23.288609 19914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:23.311444 19914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:23.315371 19914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:23.315658 19914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:23.316583 19914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:23.317862 19914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:23.317965 19914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:23.318143 19914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:23.318216 19914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:23.350208 19914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:43823
I20260619 06:32:23.350229 20041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:43823 every 8 connection(s)
I20260619 06:32:23.351490 19914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:32:23.354282 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 19914
I20260619 06:32:23.354475 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:32:23.354911 389 external_mini_cluster.cc:1428] Setting key de0d1269b7a091e689340525607fd956
Jun 19 06:32:23 dist-test-slave-4t1f krb5kdc[19674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.66@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:23.371975 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.354189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:35725 (local address 127.0.97.126:43949)
0619 06:32:23.354391 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:23.354397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:23.355638 (+ 1241us) server_negotiation.cc:407] Connection header received
0619 06:32:23.357908 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:23.357914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:23.357997 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:23.358115 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:23.359561 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.360251 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.361851 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.362278 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.362412 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.365976 (+ 3564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:23.365980 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:23.365984 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:23.366013 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:23.368865 (+ 2852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:23.369784 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:23.369788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:23.369790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:23.369919 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:23.370375 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:23.370378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:23.370380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:23.370638 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:23.370745 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:23.371562 (+ 817us) server_negotiation.cc:299] Negotiation successful
0619 06:32:23.371738 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:32:23.373773 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.354712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43949 (local address 127.0.97.66:35725)
0619 06:32:23.355485 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:23.355530 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:23.357658 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:23.358347 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:23.358355 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:23.358865 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:23.359400 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.359417 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.360390 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.360396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.361703 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.361715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.362490 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.362497 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.362718 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.363378 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:23.363399 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:23.365774 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:23.369004 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:23.369016 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:23.369033 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:23.369655 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:23.370027 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:23.370034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:23.370041 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:23.370270 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:23.370764 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:23.370774 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:23.371191 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 06:32:23.371473 (+ 282us) client_negotiation.cc:240] Negotiation successful
0619 06:32:23.372013 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":294,"threads_started":1}
I20260619 06:32:23.375197 20042 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43949
I20260619 06:32:23.375507 20042 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:23.376367 20042 heartbeater.cc:511] Master 127.0.97.126:43949 requested a full tablet report, sending...
I20260619 06:32:23.378127 19719 ts_manager.cc:194] Registered new tserver with Master: f586449d78c542aa8baae0b4f0988ea1 (127.0.97.66:43823)
I20260619 06:32:23.379318 19719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.66@KRBTEST.COM'} at 127.0.97.66:35725
WARNING: no policy specified for kudu/127.0.97.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
Entry for principal kudu/127.0.97.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab.
I20260619 06:32:23.410035 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43949
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:23.723676 20047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:23.723912 20047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:23.724077 20047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:23.731285 20047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:23.731372 20047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:23.731601 20047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:32:23.740729 20047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/krb5kdc/kudu_127.0.97.67.keytab
--principal=kudu/127.0.97.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:43949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:23.741883 20047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:23.743155 20047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:23.755899 20055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:23.755873 20053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:23.755949 20052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:23.756680 20047 server_base.cc:1034] running on GCE node
I20260619 06:32:23.757375 20047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:23.758298 20047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:23.759517 20047 hybrid_clock.cc:648] HybridClock initialized: now 1781850743759466 us; error 38 us; skew 500 ppm
I20260619 06:32:23.759920 20047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:23 dist-test-slave-4t1f krb5kdc[19674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:23.763183 20047 init.cc:377] Logged in from keytab as kudu/127.0.97.67@KRBTEST.COM (short username kudu)
I20260619 06:32:23.764550 20047 webserver.cc:460] Webserver started at http://127.0.97.67:45547/ using document root <none> and password file <none>
I20260619 06:32:23.765287 20047 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:23.765426 20047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:23.765726 20047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:23.768431 20047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "e7624a7538924fd0877a9c7315a9c04b"
format_stamp: "Formatted at 2026-06-19 06:32:23 on dist-test-slave-4t1f"
server_key: "18d6821a32310810eb286a228c308627"
server_key_iv: "e0b2116b7719d72bf8f01220f4b51a21"
server_key_version: "encryptionkey@0"
I20260619 06:32:23.769248 20047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "e7624a7538924fd0877a9c7315a9c04b"
format_stamp: "Formatted at 2026-06-19 06:32:23 on dist-test-slave-4t1f"
server_key: "18d6821a32310810eb286a228c308627"
server_key_iv: "e0b2116b7719d72bf8f01220f4b51a21"
server_key_version: "encryptionkey@0"
I20260619 06:32:23.773972 20047 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:32:23.777330 20062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:23.778609 20047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:32:23.778781 20047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "e7624a7538924fd0877a9c7315a9c04b"
format_stamp: "Formatted at 2026-06-19 06:32:23 on dist-test-slave-4t1f"
server_key: "18d6821a32310810eb286a228c308627"
server_key_iv: "e0b2116b7719d72bf8f01220f4b51a21"
server_key_version: "encryptionkey@0"
I20260619 06:32:23.779026 20047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:23.809820 20047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:23.814002 20047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:23.814214 20047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:23.814999 20047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:23.816284 20047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:23.816354 20047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:23.816437 20047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:23.816485 20047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:23.847329 20047 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:41757
I20260619 06:32:23.847375 20174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:41757 every 8 connection(s)
I20260619 06:32:23.848495 20047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:32:23.851408 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20047
I20260619 06:32:23.851575 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:32:23.852118 389 external_mini_cluster.cc:1428] Setting key 32fca830181b223ac1024008a61aac0d
Jun 19 06:32:23 dist-test-slave-4t1f krb5kdc[19674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.67@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:23.867934 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.851198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:56333 (local address 127.0.97.126:43949)
0619 06:32:23.853516 (+ 2318us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:23.853522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:23.853541 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:32:23.855342 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:23.855346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:23.855431 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:23.855507 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:23.856842 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.857665 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.859152 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.859533 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.859623 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.862719 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:23.862723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:23.862726 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:23.862755 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:23.865106 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:23.865753 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:23.865759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:23.865762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:23.865906 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:23.866329 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:23.866334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:23.866337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:23.866610 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:23.866773 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:23.867352 (+ 579us) server_negotiation.cc:299] Negotiation successful
0619 06:32:23.867559 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2200}
I20260619 06:32:23.868781 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.851794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:43949 (local address 127.0.97.67:56333)
0619 06:32:23.852636 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:23.852696 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:23.855066 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:23.855710 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:23.855718 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:23.856185 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:23.856702 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.856718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.857826 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.857830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.859025 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.859036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.859697 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.859702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.859889 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.860479 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:32:23.860499 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:23.862544 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:23.865233 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:23.865239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:23.865249 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:23.865620 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:23.866030 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:32:23.866036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:32:23.866039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:32:23.866212 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:32:23.866730 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:23.866737 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:23.867012 (+ 275us) client_negotiation.cc:769] Sending connection context
0619 06:32:23.867274 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 06:32:23.867737 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":346,"threads_started":1}
I20260619 06:32:23.870007 20175 heartbeater.cc:347] Connected to a master server at 127.0.97.126:43949
I20260619 06:32:23.870280 20175 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:23.870946 20175 heartbeater.cc:511] Master 127.0.97.126:43949 requested a full tablet report, sending...
I20260619 06:32:23.872256 19719 ts_manager.cc:194] Registered new tserver with Master: e7624a7538924fd0877a9c7315a9c04b (127.0.97.67:41757)
I20260619 06:32:23.873015 19719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.67@KRBTEST.COM'} at 127.0.97.67:56333
I20260619 06:32:23.878983 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:32:23.881282 19908 heartbeater.cc:503] Master 127.0.97.126:43949 was elected leader, sending a full tablet report...
I20260619 06:32:23.895337 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.883087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.0.97.126:43949)
0619 06:32:23.883259 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:23.883263 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:23.883717 (+ 454us) server_negotiation.cc:407] Connection header received
0619 06:32:23.883898 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:23.883903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:23.883971 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:23.884067 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:23.885418 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.886106 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.887404 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.887798 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.887919 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.890308 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:23.890314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:23.890318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:23.890355 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:23.892752 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:23.893464 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:23.893470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:23.893473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:23.893604 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:23.893974 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:23.893979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:23.893982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:23.894283 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:23.894528 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:23.894837 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 06:32:23.895050 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:32:23.898641 19719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:32:23.905682 19719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:32:23.937413 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.916611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45640 (local address 127.0.97.67:41757)
0619 06:32:23.929276 (+ 12665us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:23.929288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:23.929324 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:32:23.929432 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:23.929440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:23.929728 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:23.930650 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:23.930917 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.932292 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.934719 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.936197 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.936328 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.936537 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:23.936860 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 06:32:23.937004 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12467,"thread_start_us":12293,"threads_started":1}
I20260619 06:32:23.937491 20187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.916342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:41757 (local address 127.0.0.1:45640)
0619 06:32:23.923036 (+ 6694us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:23.923088 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:23.923241 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:23.930039 (+ 6798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:23.930051 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:23.930107 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:23.930457 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.930466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.932450 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.932457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.934539 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.934552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.936400 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.936406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.936612 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.936667 (+ 55us) client_negotiation.cc:769] Sending connection context
0619 06:32:23.936803 (+ 136us) client_negotiation.cc:240] Negotiation successful
0619 06:32:23.936984 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6477,"mutex_wait_us":1,"thread_start_us":6357,"threads_started":1}
I20260619 06:32:23.938596 20192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.928079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36795 (local address 127.0.0.1:35174)
0619 06:32:23.929687 (+ 1608us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:23.929709 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:23.929875 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:23.931550 (+ 1675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:23.931558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:23.931593 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:23.932037 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.932051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.933845 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.933849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.935812 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.935824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.937775 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.937781 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.937962 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.937986 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 06:32:23.938093 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 06:32:23.938212 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"mutex_wait_us":24,"thread_start_us":1289,"threads_started":1}
I20260619 06:32:23.938613 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.923819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.0.97.65:36795)
0619 06:32:23.930565 (+ 6746us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:23.930577 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:23.930608 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:32:23.930721 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:23.930729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:23.931024 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:23.931249 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:23.932250 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.933710 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.935962 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.937277 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.937398 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.937632 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:23.938138 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 06:32:23.938293 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6529,"thread_start_us":6319,"threads_started":1}
I20260619 06:32:23.940629 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.916562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43823 (local address 127.0.0.1:49514)
0619 06:32:23.927436 (+ 10874us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:23.927461 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:23.927598 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:23.932995 (+ 5397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:23.932999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:23.933024 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:23.933426 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.933435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.935333 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.935338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.937340 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:23.937354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.939916 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.939923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:23.940083 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.940125 (+ 42us) client_negotiation.cc:769] Sending connection context
0619 06:32:23.940195 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 06:32:23.940311 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10704,"thread_start_us":11137,"threads_started":1}
I20260619 06:32:23.940977 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:23.917111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.0.97.66:43823)
0619 06:32:23.932587 (+ 15476us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:23.932592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:23.932613 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:32:23.932688 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:23.932693 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:23.932832 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:23.933003 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:23.933626 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.935182 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.937542 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:23.939668 (+ 2126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:23.939869 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:23.940056 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:23.940418 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 06:32:23.940578 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15329,"thread_start_us":15219,"threads_started":1}
I20260619 06:32:23.941635 20110 tablet_service.cc:1467] Processing CreateTablet for tablet adaa13e539e540daba5821b00ef3ed66 (DEFAULT_TABLE table=test-table [id=618f5e646e63462a88c6d7eacff74fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:23.942219 19843 tablet_service.cc:1467] Processing CreateTablet for tablet adaa13e539e540daba5821b00ef3ed66 (DEFAULT_TABLE table=test-table [id=618f5e646e63462a88c6d7eacff74fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:23.943151 20110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adaa13e539e540daba5821b00ef3ed66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:23.944060 19843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adaa13e539e540daba5821b00ef3ed66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:23.943931 19977 tablet_service.cc:1467] Processing CreateTablet for tablet adaa13e539e540daba5821b00ef3ed66 (DEFAULT_TABLE table=test-table [id=618f5e646e63462a88c6d7eacff74fa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:23.945005 19977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adaa13e539e540daba5821b00ef3ed66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:23.955276 20193 tablet_bootstrap.cc:492] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Bootstrap starting.
I20260619 06:32:23.957368 20194 tablet_bootstrap.cc:492] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Bootstrap starting.
I20260619 06:32:23.959033 20193 tablet_bootstrap.cc:654] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:23.959062 20195 tablet_bootstrap.cc:492] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Bootstrap starting.
I20260619 06:32:23.960337 20193 log.cc:824] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:23.960636 20194 tablet_bootstrap.cc:654] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:23.961752 20194 log.cc:824] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:23.962054 20195 tablet_bootstrap.cc:654] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:23.963178 20195 log.cc:824] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:23.963212 20193 tablet_bootstrap.cc:492] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: No bootstrap required, opened a new log
I20260619 06:32:23.963462 20193 ts_tablet_manager.cc:1397] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 06:32:23.964560 20194 tablet_bootstrap.cc:492] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: No bootstrap required, opened a new log
I20260619 06:32:23.964855 20194 ts_tablet_manager.cc:1397] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260619 06:32:23.971199 20195 tablet_bootstrap.cc:492] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: No bootstrap required, opened a new log
I20260619 06:32:23.971453 20195 ts_tablet_manager.cc:1397] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260619 06:32:23.973981 20193 raft_consensus.cc:348] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:23.974406 20193 raft_consensus.cc:374] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:23.974498 20193 raft_consensus.cc:729] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f586449d78c542aa8baae0b4f0988ea1, State: Initialized, Role: FOLLOWER
I20260619 06:32:23.975101 20193 consensus_queue.cc:260] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:23.975023 20194 raft_consensus.cc:348] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:23.975399 20194 raft_consensus.cc:374] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:23.975494 20194 raft_consensus.cc:729] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7624a7538924fd0877a9c7315a9c04b, State: Initialized, Role: FOLLOWER
I20260619 06:32:23.976153 20194 consensus_queue.cc:260] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:23.977737 20175 heartbeater.cc:503] Master 127.0.97.126:43949 was elected leader, sending a full tablet report...
I20260619 06:32:23.978365 20194 ts_tablet_manager.cc:1428] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260619 06:32:23.979017 20195 raft_consensus.cc:348] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:23.979597 20042 heartbeater.cc:503] Master 127.0.97.126:43949 was elected leader, sending a full tablet report...
I20260619 06:32:23.980150 20193 ts_tablet_manager.cc:1428] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Time spent starting tablet: real 0.017s user 0.007s sys 0.009s
I20260619 06:32:23.980299 20195 raft_consensus.cc:374] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:23.980455 20195 raft_consensus.cc:729] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e08645644ef14f86b74b87991e592915, State: Initialized, Role: FOLLOWER
I20260619 06:32:23.981148 20195 consensus_queue.cc:260] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:23.983270 20195 ts_tablet_manager.cc:1428] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260619 06:32:24.053678 20199 raft_consensus.cc:482] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:32:24.053995 20199 raft_consensus.cc:504] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:24.055541 20199 leader_election.cc:288] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7624a7538924fd0877a9c7315a9c04b (127.0.97.67:41757), e08645644ef14f86b74b87991e592915 (127.0.97.65:36795)
I20260619 06:32:24.062618 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.055941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:50393 (local address 127.0.97.67:41757)
0619 06:32:24.056168 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.056177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.056670 (+ 493us) server_negotiation.cc:407] Connection header received
0619 06:32:24.056932 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.056937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.057011 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.057144 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:24.057782 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.058810 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.060781 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.061678 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.061773 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.061842 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.062232 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.062355 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 06:32:24.062734 20202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.055887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:41757 (local address 127.0.97.66:50393)
0619 06:32:24.056547 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:24.056563 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:24.056790 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:24.057182 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:24.057187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:24.057219 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:24.057645 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.057657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.058930 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.058935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.060632 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.060644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.061881 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.061888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.062066 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.062098 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 06:32:24.062194 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 06:32:24.062316 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":71,"thread_start_us":393,"threads_started":1}
I20260619 06:32:24.062857 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.056878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:44611 (local address 127.0.97.65:36795)
0619 06:32:24.057101 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.057108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.057533 (+ 425us) server_negotiation.cc:407] Connection header received
0619 06:32:24.057677 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.057682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.057765 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.057870 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:24.058439 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.059289 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.061025 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.061921 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.062054 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.062160 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.062483 (+ 323us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.062609 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 06:32:24.063107 20203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.056686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36795 (local address 127.0.97.66:44611)
0619 06:32:24.057430 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:24.057442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:24.057545 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:24.057923 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:24.057929 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:24.057952 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:24.058296 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.058310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.059423 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.059429 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.060878 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.060891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.062119 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.062124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.062311 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.062342 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:32:24.062439 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 06:32:24.062551 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":4,"thread_start_us":438,"threads_started":1}
I20260619 06:32:24.063346 20200 raft_consensus.cc:482] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:32:24.063670 20200 raft_consensus.cc:504] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:24.063938 20130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaa13e539e540daba5821b00ef3ed66" candidate_uuid: "f586449d78c542aa8baae0b4f0988ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7624a7538924fd0877a9c7315a9c04b" is_pre_election: true
I20260619 06:32:24.064185 19863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaa13e539e540daba5821b00ef3ed66" candidate_uuid: "f586449d78c542aa8baae0b4f0988ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08645644ef14f86b74b87991e592915" is_pre_election: true
I20260619 06:32:24.064459 20130 raft_consensus.cc:2454] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f586449d78c542aa8baae0b4f0988ea1 in term 0.
I20260619 06:32:24.064764 19863 raft_consensus.cc:2454] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f586449d78c542aa8baae0b4f0988ea1 in term 0.
I20260619 06:32:24.065222 19933 leader_election.cc:302] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7624a7538924fd0877a9c7315a9c04b, f586449d78c542aa8baae0b4f0988ea1; no voters:
I20260619 06:32:24.065650 20200 leader_election.cc:288] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f586449d78c542aa8baae0b4f0988ea1 (127.0.97.66:43823), e08645644ef14f86b74b87991e592915 (127.0.97.65:36795)
I20260619 06:32:24.065742 20199 raft_consensus.cc:2781] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:32:24.065896 20199 raft_consensus.cc:482] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:32:24.066000 20199 raft_consensus.cc:3037] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:24.069264 20199 raft_consensus.cc:504] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:24.069969 20199 leader_election.cc:288] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [CANDIDATE]: Term 1 election: Requested vote from peers e7624a7538924fd0877a9c7315a9c04b (127.0.97.67:41757), e08645644ef14f86b74b87991e592915 (127.0.97.65:36795)
I20260619 06:32:24.070413 19863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaa13e539e540daba5821b00ef3ed66" candidate_uuid: "f586449d78c542aa8baae0b4f0988ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08645644ef14f86b74b87991e592915"
I20260619 06:32:24.070521 20130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaa13e539e540daba5821b00ef3ed66" candidate_uuid: "f586449d78c542aa8baae0b4f0988ea1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7624a7538924fd0877a9c7315a9c04b"
I20260619 06:32:24.070611 19863 raft_consensus.cc:3037] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:24.070726 20130 raft_consensus.cc:3037] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:24.072469 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.065965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:43823 (local address 127.0.97.67:60731)
0619 06:32:24.066296 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:24.066317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:24.066448 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:24.066950 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:24.066955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:24.066981 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:24.067179 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.067195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.068565 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.068570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.069950 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.069965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.071651 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.071658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.071901 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.071936 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 06:32:24.072031 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 06:32:24.072176 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260619 06:32:24.073167 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.066059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:60731 (local address 127.0.97.66:43823)
0619 06:32:24.066239 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.066244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.066587 (+ 343us) server_negotiation.cc:407] Connection header received
0619 06:32:24.066652 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.066657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.066720 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.066795 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:24.067325 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.068441 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.070139 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.071359 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.071480 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.071566 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.072714 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.072836 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:32:24.073815 19997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaa13e539e540daba5821b00ef3ed66" candidate_uuid: "e7624a7538924fd0877a9c7315a9c04b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586449d78c542aa8baae0b4f0988ea1" is_pre_election: true
I20260619 06:32:24.074092 20130 raft_consensus.cc:2454] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f586449d78c542aa8baae0b4f0988ea1 in term 1.
I20260619 06:32:24.074363 19997 raft_consensus.cc:2379] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7624a7538924fd0877a9c7315a9c04b in current term 1: Already voted for candidate f586449d78c542aa8baae0b4f0988ea1 in this term.
I20260619 06:32:24.074507 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.066463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:45211 (local address 127.0.97.65:36795)
0619 06:32:24.066794 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.066802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.067436 (+ 634us) server_negotiation.cc:407] Connection header received
0619 06:32:24.067584 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.067588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.067657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.067743 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:24.068352 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.069401 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.071897 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.073242 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.073330 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.073430 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.074032 (+ 602us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.074194 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260619 06:32:24.075030 19933 leader_election.cc:302] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e7624a7538924fd0877a9c7315a9c04b, f586449d78c542aa8baae0b4f0988ea1; no voters:
I20260619 06:32:24.075250 20204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.066263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:36795 (local address 127.0.97.67:45211)
0619 06:32:24.067324 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:24.067344 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:24.067468 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:24.067778 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:24.067782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:24.067817 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:24.068178 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.068188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.069592 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.069597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.071692 (+ 2095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:24.071707 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.073617 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.073625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:24.073831 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.073872 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 06:32:24.074681 (+ 809us) client_negotiation.cc:240] Negotiation successful
0619 06:32:24.074836 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":769,"threads_started":1}
I20260619 06:32:24.075941 20199 raft_consensus.cc:2781] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:24.075912 19862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adaa13e539e540daba5821b00ef3ed66" candidate_uuid: "e7624a7538924fd0877a9c7315a9c04b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e08645644ef14f86b74b87991e592915" is_pre_election: true
I20260619 06:32:24.076491 19863 raft_consensus.cc:2454] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f586449d78c542aa8baae0b4f0988ea1 in term 1.
I20260619 06:32:24.076907 20066 leader_election.cc:302] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7624a7538924fd0877a9c7315a9c04b; no voters: e08645644ef14f86b74b87991e592915, f586449d78c542aa8baae0b4f0988ea1
I20260619 06:32:24.078143 20200 raft_consensus.cc:2726] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 06:32:24.078460 20199 raft_consensus.cc:686] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 1 LEADER]: Becoming Leader. State: Replica: f586449d78c542aa8baae0b4f0988ea1, State: Running, Role: LEADER
I20260619 06:32:24.079063 20199 consensus_queue.cc:237] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } }
I20260619 06:32:24.083959 19718 catalog_manager.cc:5495] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 reported cstate change: term changed from 0 to 1, leader changed from <none> to f586449d78c542aa8baae0b4f0988ea1 (127.0.97.66). New cstate: current_term: 1 leader_uuid: "f586449d78c542aa8baae0b4f0988ea1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f586449d78c542aa8baae0b4f0988ea1" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 43823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 } health_report { overall_health: UNKNOWN } } }
W20260619 06:32:24.100503 20176 tablet.cc:2369] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:32:24.103420 20043 tablet.cc:2369] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 06:32:24.103433 19909 tablet.cc:2369] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:24.127336 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.122520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49530 (local address 127.0.97.66:43823)
0619 06:32:24.122671 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.122676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.122757 (+ 81us) server_negotiation.cc:407] Connection header received
0619 06:32:24.122918 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.122922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.123009 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.123135 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:32:24.123390 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.124110 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.125459 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.125854 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.125947 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.126359 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:32:24.126823 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:32:24.126945 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.127013 (+ 68us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.127072 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:32:24.134840 19863 raft_consensus.cc:1261] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 1 FOLLOWER]: Refusing update from remote peer f586449d78c542aa8baae0b4f0988ea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:24.134907 20130 raft_consensus.cc:1261] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 1 FOLLOWER]: Refusing update from remote peer f586449d78c542aa8baae0b4f0988ea1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:24.135769 20205 consensus_queue.cc:1035] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e08645644ef14f86b74b87991e592915" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 36795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:24.136106 20199 consensus_queue.cc:1035] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7624a7538924fd0877a9c7315a9c04b" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 41757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:24.148561 20210 mvcc.cc:205] Tried to move back new op lower bound from 7298460647967109120 to 7298460647752839168. Current Snapshot: MvccSnapshot[applied={T|T < 7298460647967109120}]
I20260619 06:32:24.158772 19718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:24.159092 19718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:24.162968 19718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c47025979bdc45968557ec68f9584f96: Sending DeleteTablet for 3 replicas of tablet adaa13e539e540daba5821b00ef3ed66
I20260619 06:32:24.163968 20110 tablet_service.cc:1514] Processing DeleteTablet for tablet adaa13e539e540daba5821b00ef3ed66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:24 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:45640
I20260619 06:32:24.164278 19843 tablet_service.cc:1514] Processing DeleteTablet for tablet adaa13e539e540daba5821b00ef3ed66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:24 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:35174
I20260619 06:32:24.164711 19977 tablet_service.cc:1514] Processing DeleteTablet for tablet adaa13e539e540daba5821b00ef3ed66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:24 UTC) from {username='kudu', principal='kudu/127.0.97.126@KRBTEST.COM'} at 127.0.0.1:49514
I20260619 06:32:24.166620 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 19780
I20260619 06:32:24.181510 20219 tablet_replica.cc:330] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: stopping tablet replica
I20260619 06:32:24.181977 20219 raft_consensus.cc:2229] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:24.182448 20219 raft_consensus.cc:2258] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:24.186743 20219 ts_tablet_manager.cc:1905] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:24.190711 20220 tablet_replica.cc:330] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: stopping tablet replica
I20260619 06:32:24.191238 20220 raft_consensus.cc:2229] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:32:24.191869 20218 tablet_replica.cc:330] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: stopping tablet replica
I20260619 06:32:24.191918 20220 raft_consensus.cc:2258] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:24.192260 20218 raft_consensus.cc:2229] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:24.192840 20218 raft_consensus.cc:2258] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:24.194790 20219 ts_tablet_manager.cc:1918] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:24.194886 20218 ts_tablet_manager.cc:1905] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:24.195042 20219 log.cc:1196] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-0/wal/wals/adaa13e539e540daba5821b00ef3ed66
I20260619 06:32:24.195626 20219 ts_tablet_manager.cc:1939] T adaa13e539e540daba5821b00ef3ed66 P e08645644ef14f86b74b87991e592915: Deleting consensus metadata
I20260619 06:32:24.196794 20220 ts_tablet_manager.cc:1905] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:24.197664 19706 catalog_manager.cc:4841] TS e08645644ef14f86b74b87991e592915 (127.0.97.65:36795): tablet adaa13e539e540daba5821b00ef3ed66 (table test-table [id=618f5e646e63462a88c6d7eacff74fa4]) successfully deleted
I20260619 06:32:24.201511 20218 ts_tablet_manager.cc:1918] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:24.201668 20218 log.cc:1196] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/adaa13e539e540daba5821b00ef3ed66
I20260619 06:32:24.202149 20218 ts_tablet_manager.cc:1939] T adaa13e539e540daba5821b00ef3ed66 P e7624a7538924fd0877a9c7315a9c04b: Deleting consensus metadata
I20260619 06:32:24.203747 19706 catalog_manager.cc:4841] TS e7624a7538924fd0877a9c7315a9c04b (127.0.97.67:41757): tablet adaa13e539e540daba5821b00ef3ed66 (table test-table [id=618f5e646e63462a88c6d7eacff74fa4]) successfully deleted
I20260619 06:32:24.204452 20220 ts_tablet_manager.cc:1918] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:24.204661 20220 log.cc:1196] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/adaa13e539e540daba5821b00ef3ed66
I20260619 06:32:24.205191 20220 ts_tablet_manager.cc:1939] T adaa13e539e540daba5821b00ef3ed66 P f586449d78c542aa8baae0b4f0988ea1: Deleting consensus metadata
Jun 19 06:32:24 dist-test-slave-4t1f krb5kdc[19674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.65@KRBTEST.COM
I20260619 06:32:24.206876 19704 catalog_manager.cc:4841] TS f586449d78c542aa8baae0b4f0988ea1 (127.0.97.66:43823): tablet adaa13e539e540daba5821b00ef3ed66 (table test-table [id=618f5e646e63462a88c6d7eacff74fa4]) successfully deleted
I20260619 06:32:24.214068 20191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.167949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.0.97.65:36795)
0619 06:32:24.168192 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.168201 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.187225 (+ 19024us) server_negotiation.cc:407] Connection header received
0619 06:32:24.187746 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.187755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.187871 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.187967 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:24.192317 (+ 4350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.193469 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.200539 (+ 7070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.201306 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.201425 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.207256 (+ 5831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:24.207270 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:24.207288 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:24.207334 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:24.210994 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:24.211877 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:24.211886 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:24.211889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:24.212048 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:24.212488 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:24.212494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:24.212497 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:24.212937 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:24.213266 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.213502 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.213724 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:32:24.214722 19903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:24.304528 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 19780
I20260619 06:32:24.322165 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 19914
Jun 19 06:32:24 dist-test-slave-4t1f krb5kdc[19674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.66@KRBTEST.COM
I20260619 06:32:24.339457 20190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.322971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.97.66:43823)
0619 06:32:24.323157 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.323161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.323184 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:32:24.323383 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.323387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.323472 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.323550 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:24.325313 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.325987 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.327905 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.328408 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.328525 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.333383 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:24.333394 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:24.333408 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:24.333442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:24.336532 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:24.337333 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:24.337348 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:24.337351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:24.337506 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:24.337927 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:24.337932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:24.337941 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:24.338313 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:24.338621 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.338925 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.339150 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:32:24.340078 20037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:24.451277 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 19914
I20260619 06:32:24.469944 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20047
Jun 19 06:32:24 dist-test-slave-4t1f krb5kdc[19674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.67@KRBTEST.COM
I20260619 06:32:24.485406 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:24.470688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.0.97.67:41757)
0619 06:32:24.470883 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:24.470889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:24.470958 (+ 69us) server_negotiation.cc:407] Connection header received
0619 06:32:24.471181 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:24.471192 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:24.471257 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:24.471365 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:24.472605 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.473425 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.474739 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:24.475129 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:24.475278 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:24.479703 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:24.479717 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:24.479733 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:24.479770 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:24.482464 (+ 2694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:24.483173 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:24.483181 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:24.483183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:24.483316 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:24.483672 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:24.483678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:24.483681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:24.484028 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:24.484314 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:24.484667 (+ 353us) server_negotiation.cc:299] Negotiation successful
0619 06:32:24.484889 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:32:24.486021 20170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:24.575340 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20047
I20260619 06:32:24.591399 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 19687
I20260619 06:32:24.592367 19749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:24.700560 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 19687
2026-06-19T06:32:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4191 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T06:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:24Z Disabled control of system clock
I20260619 06:32:24.773006 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38041
--webserver_interface=127.0.97.126
--webserver_port=0
--builtin_ntp_servers=127.0.97.84:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:38041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:32:25.084230 20229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:25.084513 20229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:25.084601 20229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:25.091905 20229 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:25.092003 20229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:25.092074 20229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:25.092183 20229 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:25.092244 20229 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:25.101311 20229 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:38041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:38041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:25.102526 20229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:25.104260 20229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:25.115814 20234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:25.115782 20235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:25.115919 20237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:25.116912 20229 server_base.cc:1034] running on GCE node
I20260619 06:32:25.117707 20229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:25.118984 20229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:25.120218 20229 hybrid_clock.cc:648] HybridClock initialized: now 1781850745120188 us; error 40 us; skew 500 ppm
I20260619 06:32:25.120546 20229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:25.122658 20229 webserver.cc:460] Webserver started at http://127.0.97.126:38481/ using document root <none> and password file <none>
I20260619 06:32:25.123292 20229 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:25.123421 20229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:25.123725 20229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:25.126448 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "9bc32172bfef4a4f9f3ea882527006bf"
format_stamp: "Formatted at 2026-06-19 06:32:25 on dist-test-slave-4t1f"
server_key: "1ab52810f29fabba2249445daa727b76"
server_key_iv: "e839f0222048d4abf184023ec6f12e8b"
server_key_version: "encryptionkey@0"
I20260619 06:32:25.127290 20229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "9bc32172bfef4a4f9f3ea882527006bf"
format_stamp: "Formatted at 2026-06-19 06:32:25 on dist-test-slave-4t1f"
server_key: "1ab52810f29fabba2249445daa727b76"
server_key_iv: "e839f0222048d4abf184023ec6f12e8b"
server_key_version: "encryptionkey@0"
I20260619 06:32:25.131994 20229 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260619 06:32:25.135150 20243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:25.136402 20229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:32:25.136581 20229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "9bc32172bfef4a4f9f3ea882527006bf"
format_stamp: "Formatted at 2026-06-19 06:32:25 on dist-test-slave-4t1f"
server_key: "1ab52810f29fabba2249445daa727b76"
server_key_iv: "e839f0222048d4abf184023ec6f12e8b"
server_key_version: "encryptionkey@0"
I20260619 06:32:25.136791 20229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:25.155954 20229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:25.156695 20229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:25.156967 20229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:25.175624 20229 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:38041
I20260619 06:32:25.175691 20294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:38041 every 8 connection(s)
I20260619 06:32:25.176887 20229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:25.180341 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:25.182073 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20229
I20260619 06:32:25.182257 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:25.182809 389 external_mini_cluster.cc:1428] Setting key 309f023ad8b5819008636e778058515c
I20260619 06:32:25.193818 20295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf: Bootstrap starting.
I20260619 06:32:25.195801 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:25.184641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.0.97.126:38041)
0619 06:32:25.185639 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:25.185649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:25.185686 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:32:25.187154 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:25.187175 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:25.187598 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:25.187886 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:25.188404 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:25.189659 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:25.191573 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:25.192199 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:25.192327 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:25.193585 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:25.193592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:25.193603 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:25.194177 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:25.194262 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:25.194305 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:25.194399 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:25.194618 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 06:32:25.194917 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":430,"threads_started":1}
I20260619 06:32:25.197142 20295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:25.198307 20295 log.cc:824] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:25.200717 20295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf: No bootstrap required, opened a new log
I20260619 06:32:25.207664 20295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc32172bfef4a4f9f3ea882527006bf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38041 } }
I20260619 06:32:25.208094 20295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:25.208211 20295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc32172bfef4a4f9f3ea882527006bf, State: Initialized, Role: FOLLOWER
I20260619 06:32:25.208787 20295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc32172bfef4a4f9f3ea882527006bf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38041 } }
I20260619 06:32:25.209092 20295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:25.209244 20295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:25.209417 20295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:25.211412 20295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc32172bfef4a4f9f3ea882527006bf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38041 } }
I20260619 06:32:25.211882 20295 leader_election.cc:302] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bc32172bfef4a4f9f3ea882527006bf; no voters:
I20260619 06:32:25.212457 20295 leader_election.cc:288] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:25.212612 20300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:25.213317 20300 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [term 1 LEADER]: Becoming Leader. State: Replica: 9bc32172bfef4a4f9f3ea882527006bf, State: Running, Role: LEADER
I20260619 06:32:25.213817 20300 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc32172bfef4a4f9f3ea882527006bf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38041 } }
I20260619 06:32:25.214668 20295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:25.216568 20302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bc32172bfef4a4f9f3ea882527006bf. Latest consensus state: current_term: 1 leader_uuid: "9bc32172bfef4a4f9f3ea882527006bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc32172bfef4a4f9f3ea882527006bf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38041 } } }
I20260619 06:32:25.216892 20302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:25.217291 20301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bc32172bfef4a4f9f3ea882527006bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bc32172bfef4a4f9f3ea882527006bf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 38041 } } }
I20260619 06:32:25.217654 20301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf [sys.catalog]: This master's current role is: LEADER
W20260619 06:32:25.226225 20316 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:32:25.226362 20316 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:32:25.226526 20306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:25.228660 20306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:25.237265 20306 catalog_manager.cc:1347] Generated new cluster ID: 831b124fc67b4af1a26183228f9e2cfc
I20260619 06:32:25.237378 20306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:25.260730 20306 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:25.262540 20306 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:25.271597 20306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf: Generated new TSK 0
I20260619 06:32:25.272617 20306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:25.281869 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.65:0
--local_ip_for_outbound_sockets=127.0.97.65
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38041
--builtin_ntp_servers=127.0.97.84:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 06:32:25.604759 20319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:25.605019 20319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:25.605161 20319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:25.612311 20319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:25.612407 20319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:25.612557 20319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.65
I20260619 06:32:25.622995 20319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.65
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.65
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:25.624167 20319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:25.625869 20319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:25.638955 20325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:25.639086 20324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:25.639065 20327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:25.640431 20319 server_base.cc:1034] running on GCE node
I20260619 06:32:25.641040 20319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:25.641928 20319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:25.643162 20319 hybrid_clock.cc:648] HybridClock initialized: now 1781850745643114 us; error 60 us; skew 500 ppm
I20260619 06:32:25.643464 20319 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:25.645567 20319 webserver.cc:460] Webserver started at http://127.0.97.65:44409/ using document root <none> and password file <none>
I20260619 06:32:25.646215 20319 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:25.646348 20319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:25.646667 20319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:25.649420 20319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data/instance:
uuid: "694931bb49144ee5bc76ddbb7f6aedee"
format_stamp: "Formatted at 2026-06-19 06:32:25 on dist-test-slave-4t1f"
server_key: "4f9f70b01c7097c3d33990463689e7aa"
server_key_iv: "d6505f9f3cf6d45fb52ed214495dc0a2"
server_key_version: "encryptionkey@0"
I20260619 06:32:25.650194 20319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/wal/instance:
uuid: "694931bb49144ee5bc76ddbb7f6aedee"
format_stamp: "Formatted at 2026-06-19 06:32:25 on dist-test-slave-4t1f"
server_key: "4f9f70b01c7097c3d33990463689e7aa"
server_key_iv: "d6505f9f3cf6d45fb52ed214495dc0a2"
server_key_version: "encryptionkey@0"
I20260619 06:32:25.654899 20319 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:32:25.658324 20333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:25.659610 20319 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 06:32:25.659785 20319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/wal
uuid: "694931bb49144ee5bc76ddbb7f6aedee"
format_stamp: "Formatted at 2026-06-19 06:32:25 on dist-test-slave-4t1f"
server_key: "4f9f70b01c7097c3d33990463689e7aa"
server_key_iv: "d6505f9f3cf6d45fb52ed214495dc0a2"
server_key_version: "encryptionkey@0"
I20260619 06:32:25.660024 20319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:25.690748 20319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:25.691495 20319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:25.691731 20319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:25.692629 20319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:25.693985 20319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:25.694083 20319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:25.694152 20319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:25.694207 20319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:25.726625 20319 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.65:34951
I20260619 06:32:25.726738 20445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.65:34951 every 8 connection(s)
I20260619 06:32:25.727947 20319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/data/info.pb
I20260619 06:32:25.734603 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20319
I20260619 06:32:25.734783 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-0/wal/instance
I20260619 06:32:25.735281 389 external_mini_cluster.cc:1428] Setting key 65b55a9a365abde9f913ba6c1ca3cd80
I20260619 06:32:25.738168 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.66:0
--local_ip_for_outbound_sockets=127.0.97.66
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38041
--builtin_ntp_servers=127.0.97.84:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:32:25.742256 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:25.730448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.65:47609 (local address 127.0.97.126:38041)
0619 06:32:25.730831 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:25.730835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:25.732029 (+ 1194us) server_negotiation.cc:407] Connection header received
0619 06:32:25.734513 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:25.734518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:25.734585 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:25.734677 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:25.735969 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:25.736736 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:25.739138 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:25.739549 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:25.739647 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:25.741324 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:25.741332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:25.741339 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:25.741363 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:25.741410 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:25.741428 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:25.741511 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:25.741770 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 06:32:25.741866 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":192,"threads_started":1}
I20260619 06:32:25.743417 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:25.731057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38041 (local address 127.0.97.65:47609)
0619 06:32:25.731883 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:25.731917 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:25.734191 (+ 2274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:25.734960 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:25.734971 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:25.735085 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:25.735803 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:25.735825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:25.736871 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:25.736876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:25.738989 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:25.739014 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:25.739764 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:25.739773 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:25.740139 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:25.740914 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:25.740920 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:25.740975 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:25.741092 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:25.741533 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:25.741553 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:25.741564 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:32:25.741726 (+ 162us) client_negotiation.cc:240] Negotiation successful
0619 06:32:25.742120 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":263,"threads_started":1}
I20260619 06:32:25.744884 20446 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38041
I20260619 06:32:25.745189 20446 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:25.746007 20446 heartbeater.cc:511] Master 127.0.97.126:38041 requested a full tablet report, sending...
I20260619 06:32:25.748432 20260 ts_manager.cc:194] Registered new tserver with Master: 694931bb49144ee5bc76ddbb7f6aedee (127.0.97.65:34951)
I20260619 06:32:25.749902 20260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.65:47609
W20260619 06:32:26.081009 20450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:26.081406 20450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:26.081558 20450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:26.088999 20450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:26.089138 20450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:26.089279 20450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.66
I20260619 06:32:26.097920 20450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.66
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.66
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:26.099088 20450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:26.100723 20450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:26.113329 20456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:26.113286 20458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:26.113356 20455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:26.114236 20450 server_base.cc:1034] running on GCE node
I20260619 06:32:26.114877 20450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:26.115828 20450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:26.117043 20450 hybrid_clock.cc:648] HybridClock initialized: now 1781850746116989 us; error 53 us; skew 500 ppm
I20260619 06:32:26.117396 20450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:26.119652 20450 webserver.cc:460] Webserver started at http://127.0.97.66:34039/ using document root <none> and password file <none>
I20260619 06:32:26.120379 20450 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:26.120497 20450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:26.120808 20450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:26.123541 20450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data/instance:
uuid: "8c47f576d1ef4b6ba1e309295cc7384d"
format_stamp: "Formatted at 2026-06-19 06:32:26 on dist-test-slave-4t1f"
server_key: "b9a81e9e8c783f88c45fadb553e8d4ce"
server_key_iv: "fa8545581df240e1651d78b687ca89e2"
server_key_version: "encryptionkey@0"
I20260619 06:32:26.124274 20450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal/instance:
uuid: "8c47f576d1ef4b6ba1e309295cc7384d"
format_stamp: "Formatted at 2026-06-19 06:32:26 on dist-test-slave-4t1f"
server_key: "b9a81e9e8c783f88c45fadb553e8d4ce"
server_key_iv: "fa8545581df240e1651d78b687ca89e2"
server_key_version: "encryptionkey@0"
I20260619 06:32:26.128883 20450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 06:32:26.132002 20464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:26.133332 20450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:26.133639 20450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal
uuid: "8c47f576d1ef4b6ba1e309295cc7384d"
format_stamp: "Formatted at 2026-06-19 06:32:26 on dist-test-slave-4t1f"
server_key: "b9a81e9e8c783f88c45fadb553e8d4ce"
server_key_iv: "fa8545581df240e1651d78b687ca89e2"
server_key_version: "encryptionkey@0"
I20260619 06:32:26.133981 20450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:26.163417 20450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:26.164189 20450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:26.164458 20450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:26.165375 20450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:26.166742 20450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:26.166873 20450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:26.167021 20450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:26.167115 20450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:26.200260 20450 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.66:33293
I20260619 06:32:26.200353 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.66:33293 every 8 connection(s)
I20260619 06:32:26.201547 20450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/data/info.pb
I20260619 06:32:26.203614 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20450
I20260619 06:32:26.203786 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal/instance
I20260619 06:32:26.204268 389 external_mini_cluster.cc:1428] Setting key 938234b4a65215a2ee75879f79c2fee4
I20260619 06:32:26.207334 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.67:0
--local_ip_for_outbound_sockets=127.0.97.67
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38041
--builtin_ntp_servers=127.0.97.84:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 06:32:26.215813 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.204185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.66:60039 (local address 127.0.97.126:38041)
0619 06:32:26.204386 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.204391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.205823 (+ 1432us) server_negotiation.cc:407] Connection header received
0619 06:32:26.208368 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.208372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.208434 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.208520 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:26.209914 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.210609 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.213042 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.213477 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.213575 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.215138 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:26.215142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:26.215145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:26.215162 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:26.215210 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:26.215222 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:26.215287 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.215516 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.215592 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:32:26.217228 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.204734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38041 (local address 127.0.97.66:60039)
0619 06:32:26.205679 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:26.205719 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:26.208070 (+ 2351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:26.208800 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:26.208814 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:26.208933 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:26.209751 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.209776 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.210747 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.210755 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.212904 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.212921 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.213696 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.213705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.214019 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.214766 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:26.214772 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:26.214845 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:26.214943 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:26.215296 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:26.215309 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:26.215320 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 06:32:26.215499 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 06:32:26.215914 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":345,"threads_started":1}
I20260619 06:32:26.218626 20577 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38041
I20260619 06:32:26.218917 20577 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:26.219599 20577 heartbeater.cc:511] Master 127.0.97.126:38041 requested a full tablet report, sending...
I20260619 06:32:26.221122 20260 ts_manager.cc:194] Registered new tserver with Master: 8c47f576d1ef4b6ba1e309295cc7384d (127.0.97.66:33293)
I20260619 06:32:26.221906 20260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.66:60039
W20260619 06:32:26.523192 20580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:26.523474 20580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:26.523620 20580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:26.530560 20580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:26.530684 20580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:26.530851 20580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.67
I20260619 06:32:26.539695 20580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.67
--webserver_port=0
--tserver_master_addrs=127.0.97.126:38041
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.67
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:26.540951 20580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:26.542703 20580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:26.555168 20585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:26.555194 20586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:26.555445 20588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:26.555874 20580 server_base.cc:1034] running on GCE node
I20260619 06:32:26.556480 20580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:26.557384 20580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:26.558637 20580 hybrid_clock.cc:648] HybridClock initialized: now 1781850746558597 us; error 50 us; skew 500 ppm
I20260619 06:32:26.558944 20580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260619 06:32:26.560943 20580 webserver.cc:460] Webserver started at http://127.0.97.67:33567/ using document root <none> and password file <none>
I20260619 06:32:26.561688 20580 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:26.561813 20580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:26.562139 20580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:26.564929 20580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data/instance:
uuid: "635ebd90dd9d49d1bebbc85be2b2846d"
format_stamp: "Formatted at 2026-06-19 06:32:26 on dist-test-slave-4t1f"
server_key: "7d9e7a05c43341bddd45b00ab3407c88"
server_key_iv: "ef9ace6603db75da5f907fe266e0a0ad"
server_key_version: "encryptionkey@0"
I20260619 06:32:26.565734 20580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal/instance:
uuid: "635ebd90dd9d49d1bebbc85be2b2846d"
format_stamp: "Formatted at 2026-06-19 06:32:26 on dist-test-slave-4t1f"
server_key: "7d9e7a05c43341bddd45b00ab3407c88"
server_key_iv: "ef9ace6603db75da5f907fe266e0a0ad"
server_key_version: "encryptionkey@0"
I20260619 06:32:26.570269 20580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 06:32:26.573355 20594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:26.574543 20580 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260619 06:32:26.574708 20580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal
uuid: "635ebd90dd9d49d1bebbc85be2b2846d"
format_stamp: "Formatted at 2026-06-19 06:32:26 on dist-test-slave-4t1f"
server_key: "7d9e7a05c43341bddd45b00ab3407c88"
server_key_iv: "ef9ace6603db75da5f907fe266e0a0ad"
server_key_version: "encryptionkey@0"
I20260619 06:32:26.574908 20580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:26.588433 20580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:26.589229 20580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:26.589495 20580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:26.590384 20580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 06:32:26.591722 20580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 06:32:26.591843 20580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:26.591982 20580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 06:32:26.592070 20580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:26.626765 20580 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.67:39187
I20260619 06:32:26.626854 20706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.67:39187 every 8 connection(s)
I20260619 06:32:26.628029 20580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/data/info.pb
I20260619 06:32:26.633301 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20580
I20260619 06:32:26.633466 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal/instance
I20260619 06:32:26.633890 389 external_mini_cluster.cc:1428] Setting key 57b4502fee196b97f76f9a20996a56a2
I20260619 06:32:26.641489 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.630591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:48451 (local address 127.0.97.126:38041)
0619 06:32:26.630823 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.630831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.632403 (+ 1572us) server_negotiation.cc:407] Connection header received
0619 06:32:26.634463 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.634469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.634550 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.634689 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:26.635949 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.636899 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.638542 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.638992 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.639137 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.640546 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:26.640554 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:26.640559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:26.640586 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:26.640619 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:26.640633 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:26.640740 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.640991 (+ 251us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.641122 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:32:26.642441 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.631144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:38041 (local address 127.0.97.67:48451)
0619 06:32:26.632228 (+ 1084us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:26.632281 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:26.634168 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:26.634904 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:26.634914 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:26.635000 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:32:26.635763 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.635782 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.637055 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.637093 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.638387 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.638400 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.639232 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.639240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.639488 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.640225 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 06:32:26.640228 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:32:26.640265 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 06:32:26.640336 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:32:26.640756 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:32:26.640771 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:32:26.640785 (+ 14us) client_negotiation.cc:769] Sending connection context
0619 06:32:26.640969 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 06:32:26.641355 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":382,"threads_started":1}
I20260619 06:32:26.643661 20707 heartbeater.cc:347] Connected to a master server at 127.0.97.126:38041
I20260619 06:32:26.643923 20707 heartbeater.cc:464] Registering TS with master...
I20260619 06:32:26.644588 20707 heartbeater.cc:511] Master 127.0.97.126:38041 requested a full tablet report, sending...
I20260619 06:32:26.645918 20260 ts_manager.cc:194] Registered new tserver with Master: 635ebd90dd9d49d1bebbc85be2b2846d (127.0.97.67:39187)
I20260619 06:32:26.646724 20260 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.67:48451
I20260619 06:32:26.648793 389 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 06:32:26.657774 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.652634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54664 (local address 127.0.97.126:38041)
0619 06:32:26.652783 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.652787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.653255 (+ 468us) server_negotiation.cc:407] Connection header received
0619 06:32:26.653374 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.653377 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.653442 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.653518 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:26.653924 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.654566 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.655804 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.656201 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.656291 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.657217 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:26.657222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:26.657225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:26.657240 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:26.657268 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:26.657280 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:26.657347 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.657500 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.657571 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:32:26.660965 20260 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 06:32:26.667755 20260 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 06:32:26.690928 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.678432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.67:39187 (local address 127.0.0.1:50848)
0619 06:32:26.681042 (+ 2610us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:26.681110 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:26.681266 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:26.683596 (+ 2330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:26.683600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:26.683618 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:26.684053 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.684070 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.685619 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.685624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.688193 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.688202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.689918 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.689924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.690204 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.690250 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 06:32:26.690407 (+ 157us) client_negotiation.cc:240] Negotiation successful
0619 06:32:26.690548 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2379,"mutex_wait_us":185,"thread_start_us":2250,"threads_started":1}
I20260619 06:32:26.691009 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.679137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50848 (local address 127.0.97.67:39187)
0619 06:32:26.680469 (+ 1332us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.680480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.681217 (+ 737us) server_negotiation.cc:407] Connection header received
0619 06:32:26.682786 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.682793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.683038 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.683223 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:26.684286 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.685457 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.688412 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.689363 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.689508 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.689798 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.690462 (+ 664us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.690641 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1111,"thread_start_us":922,"threads_started":1}
I20260619 06:32:26.691056 20720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.678469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48486)
0619 06:32:26.681842 (+ 3373us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:26.681863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:26.682000 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:26.683133 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:26.683143 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:26.683183 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:26.683527 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.683548 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.685376 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.685381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.688173 (+ 2792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.688192 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.690236 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.690242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.690470 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.690501 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:32:26.690605 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 06:32:26.690717 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3222,"mutex_wait_us":124,"thread_start_us":3077,"threads_started":1}
I20260619 06:32:26.691385 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.678549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.97.65:34951)
0619 06:32:26.679189 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.679198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.682143 (+ 2945us) server_negotiation.cc:407] Connection header received
0619 06:32:26.682280 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.682290 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.682618 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.682839 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:26.683730 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.685252 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.688388 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.689781 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.689918 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.690145 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.690673 (+ 528us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.690828 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":1125,"threads_started":1}
I20260619 06:32:26.692045 20723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.681454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33293 (local address 127.0.0.1:42426)
0619 06:32:26.683176 (+ 1722us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:26.683193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:26.683298 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:26.683946 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:26.683950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:26.683977 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:26.684384 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.684400 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.686369 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.686373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.687877 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.687889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.689959 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.689966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.691467 (+ 1501us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.691507 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 06:32:26.691610 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 06:32:26.691723 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1591,"mutex_wait_us":1289,"thread_start_us":1460,"threads_started":1}
I20260619 06:32:26.692898 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.681776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.97.66:33293)
0619 06:32:26.683449 (+ 1673us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.683457 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.683483 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:32:26.683572 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.683581 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.683781 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.683952 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:26.684581 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.686202 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.688054 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.689424 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.689565 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.689817 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.692365 (+ 2548us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.692539 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1516,"thread_start_us":1632,"threads_started":1}
I20260619 06:32:26.694761 20642 tablet_service.cc:1467] Processing CreateTablet for tablet 15b9c5622a554dc4aba7a295f671f61f (DEFAULT_TABLE table=test-table [id=8bf4c873457e4716b609243f12b1f383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:26.694710 20381 tablet_service.cc:1467] Processing CreateTablet for tablet 15b9c5622a554dc4aba7a295f671f61f (DEFAULT_TABLE table=test-table [id=8bf4c873457e4716b609243f12b1f383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:26.696090 20381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15b9c5622a554dc4aba7a295f671f61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:26.696110 20642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15b9c5622a554dc4aba7a295f671f61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:26.696260 20512 tablet_service.cc:1467] Processing CreateTablet for tablet 15b9c5622a554dc4aba7a295f671f61f (DEFAULT_TABLE table=test-table [id=8bf4c873457e4716b609243f12b1f383]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 06:32:26.697474 20512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15b9c5622a554dc4aba7a295f671f61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:26.706779 20725 tablet_bootstrap.cc:492] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Bootstrap starting.
I20260619 06:32:26.710210 20726 tablet_bootstrap.cc:492] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: Bootstrap starting.
I20260619 06:32:26.710280 20725 tablet_bootstrap.cc:654] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:26.711648 20725 log.cc:824] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:26.711728 20727 tablet_bootstrap.cc:492] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Bootstrap starting.
I20260619 06:32:26.713702 20726 tablet_bootstrap.cc:654] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:26.714766 20727 tablet_bootstrap.cc:654] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:26.714869 20726 log.cc:824] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:26.715426 20725 tablet_bootstrap.cc:492] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: No bootstrap required, opened a new log
I20260619 06:32:26.715729 20725 ts_tablet_manager.cc:1397] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260619 06:32:26.715920 20727 log.cc:824] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:26.718210 20726 tablet_bootstrap.cc:492] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: No bootstrap required, opened a new log
I20260619 06:32:26.718477 20726 ts_tablet_manager.cc:1397] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260619 06:32:26.718695 20727 tablet_bootstrap.cc:492] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: No bootstrap required, opened a new log
I20260619 06:32:26.718921 20727 ts_tablet_manager.cc:1397] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 06:32:26.725736 20727 raft_consensus.cc:348] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.726154 20727 raft_consensus.cc:374] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:26.726257 20727 raft_consensus.cc:729] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c47f576d1ef4b6ba1e309295cc7384d, State: Initialized, Role: FOLLOWER
I20260619 06:32:26.726512 20725 raft_consensus.cc:348] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.726898 20725 raft_consensus.cc:374] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:26.726982 20725 raft_consensus.cc:729] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 635ebd90dd9d49d1bebbc85be2b2846d, State: Initialized, Role: FOLLOWER
I20260619 06:32:26.726890 20727 consensus_queue.cc:260] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.727521 20725 consensus_queue.cc:260] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.729472 20726 raft_consensus.cc:348] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.730027 20726 raft_consensus.cc:374] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:26.730158 20726 raft_consensus.cc:729] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 694931bb49144ee5bc76ddbb7f6aedee, State: Initialized, Role: FOLLOWER
I20260619 06:32:26.730944 20726 consensus_queue.cc:260] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.739392 20726 ts_tablet_manager.cc:1428] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20260619 06:32:26.739820 20446 heartbeater.cc:503] Master 127.0.97.126:38041 was elected leader, sending a full tablet report...
I20260619 06:32:26.748427 20577 heartbeater.cc:503] Master 127.0.97.126:38041 was elected leader, sending a full tablet report...
I20260619 06:32:26.748534 20707 heartbeater.cc:503] Master 127.0.97.126:38041 was elected leader, sending a full tablet report...
I20260619 06:32:26.748746 20725 ts_tablet_manager.cc:1428] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Time spent starting tablet: real 0.033s user 0.021s sys 0.011s
I20260619 06:32:26.749792 20727 ts_tablet_manager.cc:1428] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Time spent starting tablet: real 0.031s user 0.013s sys 0.018s
I20260619 06:32:26.844990 20732 raft_consensus.cc:482] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 06:32:26.845278 20732 raft_consensus.cc:504] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.846903 20732 leader_election.cc:288] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 694931bb49144ee5bc76ddbb7f6aedee (127.0.97.65:34951), 8c47f576d1ef4b6ba1e309295cc7384d (127.0.97.66:33293)
I20260619 06:32:26.854030 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.847664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:41863 (local address 127.0.97.66:33293)
0619 06:32:26.847902 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.847908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.847931 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:32:26.848009 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.848014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.848073 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.848193 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:26.848602 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.849618 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.852125 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.853269 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.853397 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.853498 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.853689 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.853812 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 06:32:26.854343 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.847168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.66:33293 (local address 127.0.97.67:41863)
0619 06:32:26.847364 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:26.847386 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:26.847515 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:26.848248 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:26.848253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:26.848280 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:26.848416 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.848445 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.849729 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.849732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.851981 (+ 2249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.851995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.853399 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.853403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.853523 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.853557 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 06:32:26.853628 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 06:32:26.853783 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":85}
I20260619 06:32:26.854430 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.847364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.67:34245 (local address 127.0.97.65:34951)
0619 06:32:26.847614 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.847622 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.848128 (+ 506us) server_negotiation.cc:407] Connection header received
0619 06:32:26.848284 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.848289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.848381 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.848448 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 06:32:26.848985 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.849962 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.852355 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.853566 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.853702 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.853782 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.854063 (+ 281us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.854199 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 06:32:26.854460 20734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.847279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.97.67:34245)
0619 06:32:26.848024 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:26.848036 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 06:32:26.848135 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:32:26.848494 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:32:26.848500 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:32:26.848526 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 06:32:26.848830 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.848838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.850099 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.850105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.852203 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:32:26.852212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.853734 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.853739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:32:26.853888 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.853919 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 06:32:26.854006 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 06:32:26.854137 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":1,"thread_start_us":483,"threads_started":1}
I20260619 06:32:26.855448 20532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b9c5622a554dc4aba7a295f671f61f" candidate_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" is_pre_election: true
I20260619 06:32:26.855465 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b9c5622a554dc4aba7a295f671f61f" candidate_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694931bb49144ee5bc76ddbb7f6aedee" is_pre_election: true
I20260619 06:32:26.856014 20532 raft_consensus.cc:2454] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635ebd90dd9d49d1bebbc85be2b2846d in term 0.
I20260619 06:32:26.856009 20401 raft_consensus.cc:2454] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 635ebd90dd9d49d1bebbc85be2b2846d in term 0.
I20260619 06:32:26.856737 20596 leader_election.cc:302] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 635ebd90dd9d49d1bebbc85be2b2846d, 8c47f576d1ef4b6ba1e309295cc7384d; no voters:
I20260619 06:32:26.857208 20732 raft_consensus.cc:2781] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 06:32:26.857358 20732 raft_consensus.cc:482] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 06:32:26.857483 20732 raft_consensus.cc:3037] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:26.859620 20732 raft_consensus.cc:504] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.860186 20732 leader_election.cc:288] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [CANDIDATE]: Term 1 election: Requested vote from peers 694931bb49144ee5bc76ddbb7f6aedee (127.0.97.65:34951), 8c47f576d1ef4b6ba1e309295cc7384d (127.0.97.66:33293)
I20260619 06:32:26.860630 20532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b9c5622a554dc4aba7a295f671f61f" candidate_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c47f576d1ef4b6ba1e309295cc7384d"
I20260619 06:32:26.860648 20401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15b9c5622a554dc4aba7a295f671f61f" candidate_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "694931bb49144ee5bc76ddbb7f6aedee"
I20260619 06:32:26.860807 20532 raft_consensus.cc:3037] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:26.860880 20401 raft_consensus.cc:3037] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:26.863005 20532 raft_consensus.cc:2454] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635ebd90dd9d49d1bebbc85be2b2846d in term 1.
I20260619 06:32:26.863169 20401 raft_consensus.cc:2454] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 635ebd90dd9d49d1bebbc85be2b2846d in term 1.
I20260619 06:32:26.863436 20596 leader_election.cc:302] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 635ebd90dd9d49d1bebbc85be2b2846d, 8c47f576d1ef4b6ba1e309295cc7384d; no voters:
I20260619 06:32:26.863715 20732 raft_consensus.cc:2781] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:26.864166 20732 raft_consensus.cc:686] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 1 LEADER]: Becoming Leader. State: Replica: 635ebd90dd9d49d1bebbc85be2b2846d, State: Running, Role: LEADER
I20260619 06:32:26.864692 20732 consensus_queue.cc:237] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } }
I20260619 06:32:26.868964 20259 catalog_manager.cc:5495] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d reported cstate change: term changed from 0 to 1, leader changed from <none> to 635ebd90dd9d49d1bebbc85be2b2846d (127.0.97.67). New cstate: current_term: 1 leader_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "635ebd90dd9d49d1bebbc85be2b2846d" member_type: VOTER last_known_addr { host: "127.0.97.67" port: 39187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 } health_report { overall_health: UNKNOWN } } }
W20260619 06:32:26.879699 20708 tablet.cc:2369] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:32:26.928289 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.922800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.97.67:39187)
0619 06:32:26.922979 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.922983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.923000 (+ 17us) server_negotiation.cc:407] Connection header received
0619 06:32:26.923125 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.923128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.923202 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.923322 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:32:26.923835 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.924604 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.926167 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.926575 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.926668 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.927228 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:32:26.927713 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:32:26.927838 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.927945 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.928012 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 06:32:26.936528 20401 raft_consensus.cc:1261] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 1 FOLLOWER]: Refusing update from remote peer 635ebd90dd9d49d1bebbc85be2b2846d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:26.936538 20532 raft_consensus.cc:1261] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 1 FOLLOWER]: Refusing update from remote peer 635ebd90dd9d49d1bebbc85be2b2846d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:32:26.937554 20735 consensus_queue.cc:1035] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c47f576d1ef4b6ba1e309295cc7384d" member_type: VOTER last_known_addr { host: "127.0.97.66" port: 33293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:26.937999 20732 consensus_queue.cc:1035] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [LEADER]: Connected to new peer: Peer: permanent_uuid: "694931bb49144ee5bc76ddbb7f6aedee" member_type: VOTER last_known_addr { host: "127.0.97.65" port: 34951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:32:26.949687 20738 mvcc.cc:205] Tried to move back new op lower bound from 7298460659441037312 to 7298460659161853952. Current Snapshot: MvccSnapshot[applied={T|T < 7298460659441037312}]
I20260619 06:32:26.951457 20740 mvcc.cc:205] Tried to move back new op lower bound from 7298460659441037312 to 7298460659161853952. Current Snapshot: MvccSnapshot[applied={T|T < 7298460659441037312}]
I20260619 06:32:26.963927 20259 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:26.964246 20259 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 06:32:26.967860 20259 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bc32172bfef4a4f9f3ea882527006bf: Sending DeleteTablet for 3 replicas of tablet 15b9c5622a554dc4aba7a295f671f61f
I20260619 06:32:26.969162 20512 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b9c5622a554dc4aba7a295f671f61f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:26 UTC) from {username='slave'} at 127.0.0.1:42426
I20260619 06:32:26.969162 20381 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b9c5622a554dc4aba7a295f671f61f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:26 UTC) from {username='slave'} at 127.0.0.1:48486
I20260619 06:32:26.969961 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20319
I20260619 06:32:26.970352 20642 tablet_service.cc:1514] Processing DeleteTablet for tablet 15b9c5622a554dc4aba7a295f671f61f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 06:32:26 UTC) from {username='slave'} at 127.0.0.1:50848
I20260619 06:32:26.980180 20750 tablet_replica.cc:330] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: stopping tablet replica
I20260619 06:32:26.980593 20750 raft_consensus.cc:2229] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 1 LEADER]: Raft consensus shutting down.
I20260619 06:32:26.982512 20750 raft_consensus.cc:2258] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:26.983440 20748 tablet_replica.cc:330] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: stopping tablet replica
I20260619 06:32:26.985301 20750 ts_tablet_manager.cc:1905] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:26.989398 20748 raft_consensus.cc:2229] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:26.989883 20748 raft_consensus.cc:2258] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:26.990854 20747 tablet_replica.cc:330] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: stopping tablet replica
I20260619 06:32:26.991261 20747 raft_consensus.cc:2229] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 06:32:26.991724 20747 raft_consensus.cc:2258] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 06:32:26.992136 20748 ts_tablet_manager.cc:1905] T 15b9c5622a554dc4aba7a295f671f61f P 694931bb49144ee5bc76ddbb7f6aedee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:26.993686 20721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:26.972184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.0.97.65:34951)
0619 06:32:26.972543 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:26.972553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:26.981937 (+ 9384us) server_negotiation.cc:407] Connection header received
0619 06:32:26.982128 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:26.982137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:26.982269 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:26.982407 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:26.984448 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.986016 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.988641 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:26.989555 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:26.989682 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:26.991986 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:26.991996 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:26.992012 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:26.992754 (+ 742us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:26.992811 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:26.992841 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:26.992950 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:26.993203 (+ 253us) server_negotiation.cc:299] Negotiation successful
0619 06:32:26.993317 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 06:32:26.993891 20747 ts_tablet_manager.cc:1905] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 06:32:26.994346 20441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:26.997673 20750 ts_tablet_manager.cc:1918] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:26.997879 20750 log.cc:1196] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-2/wal/wals/15b9c5622a554dc4aba7a295f671f61f
I20260619 06:32:26.998518 20750 ts_tablet_manager.cc:1939] T 15b9c5622a554dc4aba7a295f671f61f P 635ebd90dd9d49d1bebbc85be2b2846d: Deleting consensus metadata
I20260619 06:32:27.000360 20245 catalog_manager.cc:4841] TS 635ebd90dd9d49d1bebbc85be2b2846d (127.0.97.67:39187): tablet 15b9c5622a554dc4aba7a295f671f61f (table test-table [id=8bf4c873457e4716b609243f12b1f383]) successfully deleted
I20260619 06:32:27.003515 20747 ts_tablet_manager.cc:1918] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 06:32:27.003655 20747 log.cc:1196] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Deleting WAL directory at /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781850433974487-389-0/minicluster-data/ts-1/wal/wals/15b9c5622a554dc4aba7a295f671f61f
I20260619 06:32:27.004130 20747 ts_tablet_manager.cc:1939] T 15b9c5622a554dc4aba7a295f671f61f P 8c47f576d1ef4b6ba1e309295cc7384d: Deleting consensus metadata
I20260619 06:32:27.005743 20245 catalog_manager.cc:4841] TS 8c47f576d1ef4b6ba1e309295cc7384d (127.0.97.66:33293): tablet 15b9c5622a554dc4aba7a295f671f61f (table test-table [id=8bf4c873457e4716b609243f12b1f383]) successfully deleted
I20260619 06:32:27.089604 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20319
W20260619 06:32:27.107226 20244 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.65:34951 (error 108)
I20260619 06:32:27.107919 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20450
I20260619 06:32:27.109354 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.108785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48500)
0619 06:32:27.109040 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.109186 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260619 06:32:27.109599 20244 catalog_manager.cc:4584] TS 694931bb49144ee5bc76ddbb7f6aedee (127.0.97.65:34951): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 15b9c5622a554dc4aba7a295f671f61f: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
I20260619 06:32:27.118234 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.108836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.97.66:33293)
0619 06:32:27.109367 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:27.109371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:27.109386 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:32:27.109475 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:27.109482 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:27.109576 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:27.109710 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:27.110123 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:27.110862 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:27.112165 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:27.112585 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:27.112704 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:27.116692 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:27.116704 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:27.116722 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:27.117501 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:27.117562 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:27.117587 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:27.117703 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:27.117806 (+ 103us) server_negotiation.cc:299] Negotiation successful
0619 06:32:27.117911 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260619 06:32:27.118947 20572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:27.163187 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.162747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48508)
0619 06:32:27.162937 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.163038 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:32:27.164242 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.163885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48514)
0619 06:32:27.164062 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.164125 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:32:27.224797 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.224265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48528)
0619 06:32:27.224498 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.224604 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:32:27.225987 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.225601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48530)
0619 06:32:27.225763 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.225845 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 06:32:27.246166 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20450
I20260619 06:32:27.262328 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20580
I20260619 06:32:27.270345 20722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.263115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.0.97.67:39187)
0619 06:32:27.263305 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:27.263309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:27.263327 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:32:27.263523 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:27.263527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:27.263607 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:27.263689 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:27.264204 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:27.265011 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:27.267070 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:27.267517 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:27.267626 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:27.268808 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:27.268818 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:27.268832 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:27.269520 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:27.269572 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:27.269602 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:27.269684 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:27.269867 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 06:32:27.269953 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:32:27.271036 20702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:27.306865 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.306302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48536)
0619 06:32:27.306545 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.306674 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260619 06:32:27.308022 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.307588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48548)
0619 06:32:27.307792 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.307874 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 06:32:27.391831 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20580
I20260619 06:32:27.410493 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20229
I20260619 06:32:27.411679 20290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:27.520467 20719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:27.519889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.65:34951 (local address 127.0.0.1:48556)
0619 06:32:27.520098 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:32:27.520220 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.65:34951: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 06:32:27.520660 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20229
2026-06-19T06:32:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2819 ms)
[----------] 6 tests from EncryptionPolicyTest (21637 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:27 dist-test-slave-4t1f krb5kdc[20758](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:27 dist-test-slave-4t1f krb5kdc[20758](info): set up 2 sockets
Jun 19 06:32:27 dist-test-slave-4t1f krb5kdc[20758](info): commencing operation
krb5kdc: starting...
W20260619 06:32:28.621822 389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:28 dist-test-slave-4t1f krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:28.858767 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:29.176389 20771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:29.176669 20771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:29.176760 20771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:29.183877 20771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:29.184015 20771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:29.184062 20771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:29.184125 20771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:29.184170 20771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:29.194617 20771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:29.195832 20771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:29.197194 20771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:29.208950 20777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:29.209501 20776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:29.209098 20779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:29.210707 20771 server_base.cc:1034] running on GCE node
I20260619 06:32:29.211418 20771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:29.212710 20771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:29.213963 20771 hybrid_clock.cc:648] HybridClock initialized: now 1781850749213921 us; error 31 us; skew 500 ppm
I20260619 06:32:29.214296 20771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:29 dist-test-slave-4t1f krb5kdc[20758](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850749, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:29.217402 20771 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:29.218890 20771 webserver.cc:460] Webserver started at http://127.0.0.1:41021/ using document root <none> and password file <none>
I20260619 06:32:29.219607 20771 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:29.219748 20771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:29.220041 20771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:29.222913 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "ecb9477885ab4f2abe52a4295e69bd9a"
format_stamp: "Formatted at 2026-06-19 06:32:29 on dist-test-slave-4t1f"
server_key: "39520c3e001894f1adfbde760caa7f83"
server_key_iv: "9193e29e557eada7d3a6e159bc1b3c31"
server_key_version: "encryptionkey@0"
I20260619 06:32:29.223639 20771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "ecb9477885ab4f2abe52a4295e69bd9a"
format_stamp: "Formatted at 2026-06-19 06:32:29 on dist-test-slave-4t1f"
server_key: "39520c3e001894f1adfbde760caa7f83"
server_key_iv: "9193e29e557eada7d3a6e159bc1b3c31"
server_key_version: "encryptionkey@0"
I20260619 06:32:29.228463 20771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:32:29.231743 20786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:29.232959 20771 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:32:29.233157 20771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "ecb9477885ab4f2abe52a4295e69bd9a"
format_stamp: "Formatted at 2026-06-19 06:32:29 on dist-test-slave-4t1f"
server_key: "39520c3e001894f1adfbde760caa7f83"
server_key_iv: "9193e29e557eada7d3a6e159bc1b3c31"
server_key_version: "encryptionkey@0"
I20260619 06:32:29.233350 20771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:29.250118 20771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:29.253839 20771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:29.254112 20771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:29.272338 20771 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42839
I20260619 06:32:29.272377 20837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42839 every 8 connection(s)
I20260619 06:32:29.273751 20771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:29.277335 20838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:29.280694 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20771
I20260619 06:32:29.280918 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:29.281724 389 external_mini_cluster.cc:1428] Setting key 137826142a32bedb87d1f45c268055a9
I20260619 06:32:29.289944 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a: Bootstrap starting.
I20260619 06:32:29.293576 20838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:29.294845 20838 log.cc:824] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:32:29 dist-test-slave-4t1f krb5kdc[20758](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:32:29.297712 20838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a: No bootstrap required, opened a new log
I20260619 06:32:29.304484 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:29.283584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54393 (local address 127.0.0.1:42839)
0619 06:32:29.284344 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:29.284354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:29.284392 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:32:29.286034 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:29.286056 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:29.286452 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:29.286753 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:29.288448 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:29.289764 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:29.291503 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:29.292148 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:29.292274 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:29.297255 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:29.297263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:29.297275 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:29.297308 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:29.300415 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:29.301225 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:29.301232 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:29.301238 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:29.301415 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:29.301902 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:29.301906 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:29.301908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:29.302385 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:29.302748 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:29.303221 (+ 473us) server_negotiation.cc:299] Negotiation successful
0619 06:32:29.303520 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":381,"threads_started":1}
I20260619 06:32:29.305431 20838 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42839 } }
I20260619 06:32:29.305781 20838 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:29.305938 20838 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb9477885ab4f2abe52a4295e69bd9a, State: Initialized, Role: FOLLOWER
I20260619 06:32:29.306447 20838 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42839 } }
I20260619 06:32:29.306699 20838 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:29.306861 20838 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:29.307052 20838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:29.308533 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:29.309867 20838 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42839 } }
I20260619 06:32:29.310389 20838 leader_election.cc:302] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecb9477885ab4f2abe52a4295e69bd9a; no voters:
I20260619 06:32:29.310964 20838 leader_election.cc:288] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:29.311079 20843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:29.313090 20843 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [term 1 LEADER]: Becoming Leader. State: Replica: ecb9477885ab4f2abe52a4295e69bd9a, State: Running, Role: LEADER
I20260619 06:32:29.313630 20843 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42839 } }
I20260619 06:32:29.313941 20838 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:29.316280 20844 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42839 } } }
I20260619 06:32:29.316735 20844 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:29.317632 20845 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecb9477885ab4f2abe52a4295e69bd9a. Latest consensus state: current_term: 1 leader_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecb9477885ab4f2abe52a4295e69bd9a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42839 } } }
I20260619 06:32:29.318084 20845 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:29.319314 20856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:29.328755 20856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:29.343318 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:29.320016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55239 (local address 127.0.0.1:42839)
0619 06:32:29.320258 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:29.320268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:29.322571 (+ 2303us) server_negotiation.cc:407] Connection header received
0619 06:32:29.322812 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:29.322820 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:29.322928 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:29.323040 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:29.325677 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:29.326936 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:29.328922 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:29.329701 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:29.329836 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:29.333531 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:29.333539 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:29.333544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:29.333585 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:29.337638 (+ 4053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:29.339029 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:29.339038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:29.339044 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:29.339238 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:29.340041 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:29.340050 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:29.340056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:29.340600 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:29.340947 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:29.342477 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0619 06:32:29.342738 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 06:32:29.355448 20856 catalog_manager.cc:1347] Generated new cluster ID: 0df8fb03a9bf435181871b0fb516ab76
I20260619 06:32:29.355587 20856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:29.371665 20856 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:29.373234 20856 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:29.381014 20856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecb9477885ab4f2abe52a4295e69bd9a: Generated new TSK 0
I20260619 06:32:29.382025 20856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:29.393127 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20771
I20260619 06:32:29.394536 20833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:29.485539 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20771
W20260619 06:32:29.502427 20756 connection.cc:535] client connection to 127.0.0.1:42839 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42839): Connection reset by peer (error 104)
2026-06-19T06:32:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:29 dist-test-slave-4t1f krb5kdc[20870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:29 dist-test-slave-4t1f krb5kdc[20870](info): set up 2 sockets
Jun 19 06:32:29 dist-test-slave-4t1f krb5kdc[20870](info): commencing operation
krb5kdc: starting...
W20260619 06:32:30.609445 389 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:30 dist-test-slave-4t1f krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:30.840318 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:31.153779 20883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:31.154095 20883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:31.154284 20883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:31.161348 20883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:31.161492 20883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:31.161602 20883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:31.161716 20883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:31.161804 20883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:31.171653 20883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:31.172955 20883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:31.174322 20883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:31.186429 20891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:31.186453 20888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:31.186479 20889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:31.187201 20883 server_base.cc:1034] running on GCE node
I20260619 06:32:31.187914 20883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:31.189230 20883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:31.190455 20883 hybrid_clock.cc:648] HybridClock initialized: now 1781850751190419 us; error 47 us; skew 500 ppm
I20260619 06:32:31.190781 20883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:31 dist-test-slave-4t1f krb5kdc[20870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:31.193699 20883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:31.194965 20883 webserver.cc:460] Webserver started at http://127.0.0.1:35851/ using document root <none> and password file <none>
I20260619 06:32:31.195566 20883 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:31.195660 20883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:31.195891 20883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:31.198719 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "ec3b53f3bccd4282a4c8ea4b6e766414"
format_stamp: "Formatted at 2026-06-19 06:32:31 on dist-test-slave-4t1f"
server_key: "d77a41b1cbe215efc392f98cad6e7ae7"
server_key_iv: "5893e9951fc06ce4e34edf623a28b5be"
server_key_version: "encryptionkey@0"
I20260619 06:32:31.199378 20883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "ec3b53f3bccd4282a4c8ea4b6e766414"
format_stamp: "Formatted at 2026-06-19 06:32:31 on dist-test-slave-4t1f"
server_key: "d77a41b1cbe215efc392f98cad6e7ae7"
server_key_iv: "5893e9951fc06ce4e34edf623a28b5be"
server_key_version: "encryptionkey@0"
I20260619 06:32:31.204005 20883 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:32:31.207072 20898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:31.208316 20883 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:32:31.208447 20883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "ec3b53f3bccd4282a4c8ea4b6e766414"
format_stamp: "Formatted at 2026-06-19 06:32:31 on dist-test-slave-4t1f"
server_key: "d77a41b1cbe215efc392f98cad6e7ae7"
server_key_iv: "5893e9951fc06ce4e34edf623a28b5be"
server_key_version: "encryptionkey@0"
I20260619 06:32:31.208592 20883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:31.223655 20883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:31.227186 20883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:31.227388 20883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:31.245368 20883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39801
I20260619 06:32:31.245409 20949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39801 every 8 connection(s)
I20260619 06:32:31.246670 20883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:31.250212 20950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:31.252388 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20883
I20260619 06:32:31.252560 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:31.253044 389 external_mini_cluster.cc:1428] Setting key fd506b9be1c83fc5e9b8d3a6874450cd
I20260619 06:32:31.262635 20950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414: Bootstrap starting.
I20260619 06:32:31.266165 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:31.254736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59661 (local address 127.0.0.1:39801)
0619 06:32:31.255506 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:31.255517 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:31.255558 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:32:31.257262 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:31.257282 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:31.257649 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:31.257941 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:31.258497 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:31.259813 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:31.261857 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:31.262494 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:31.262613 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:31.263940 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:31.263947 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:31.263959 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:31.264491 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:31.264574 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:31.264598 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:31.264709 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:31.265045 (+ 336us) server_negotiation.cc:299] Negotiation successful
0619 06:32:31.265262 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":374,"threads_started":1}
I20260619 06:32:31.266271 20950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:31.267436 20950 log.cc:824] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:31.270152 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:31.270747 20950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414: No bootstrap required, opened a new log
I20260619 06:32:31.281315 20950 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39801 } }
I20260619 06:32:31.281829 20950 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:31.281980 20950 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3b53f3bccd4282a4c8ea4b6e766414, State: Initialized, Role: FOLLOWER
I20260619 06:32:31.282091 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:31.275207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44307 (local address 127.0.0.1:39801)
0619 06:32:31.275401 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:31.275408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:31.275840 (+ 432us) server_negotiation.cc:407] Connection header received
0619 06:32:31.275960 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:31.275966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:31.276045 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:31.276197 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:31.276635 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:31.277901 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:31.279176 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:31.279959 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:31.280109 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:31.281171 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:31.281178 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:31.281183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:31.281203 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:31.281266 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:31.281292 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:31.281436 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:31.281605 (+ 169us) server_negotiation.cc:299] Negotiation successful
0619 06:32:31.281668 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:32:31.282655 20950 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39801 } }
I20260619 06:32:31.282922 20950 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:31.283097 20950 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:31.283295 20950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:31.285933 20950 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39801 } }
I20260619 06:32:31.286443 20950 leader_election.cc:302] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec3b53f3bccd4282a4c8ea4b6e766414; no voters:
I20260619 06:32:31.287036 20950 leader_election.cc:288] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:31.287207 20960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:31.287902 20960 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [term 1 LEADER]: Becoming Leader. State: Replica: ec3b53f3bccd4282a4c8ea4b6e766414, State: Running, Role: LEADER
I20260619 06:32:31.288404 20960 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39801 } }
I20260619 06:32:31.289364 20950 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:31.291962 20961 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39801 } } }
I20260619 06:32:31.291950 20962 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec3b53f3bccd4282a4c8ea4b6e766414. Latest consensus state: current_term: 1 leader_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec3b53f3bccd4282a4c8ea4b6e766414" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39801 } } }
I20260619 06:32:31.292380 20961 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:31.292443 20962 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:31.297549 20967 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:31.299906 20977 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:32:31.302311 20967 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:31.310376 20967 catalog_manager.cc:1347] Generated new cluster ID: a8ab2851b2b14d1884310e45c39f8701
I20260619 06:32:31.310515 20967 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:31.331715 20967 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:31.333348 20967 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:31.344224 20967 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec3b53f3bccd4282a4c8ea4b6e766414: Generated new TSK 0
I20260619 06:32:31.345115 20967 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:31.362727 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20883
W20260619 06:32:31.363792 20945 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59661
W20260619 06:32:31.364503 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20883: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:31.364692 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20883
W20260619 06:32:31.381047 20868 connection.cc:535] client connection to 127.0.0.1:39801 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39801): Connection reset by peer (error 104)
2026-06-19T06:32:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:31 dist-test-slave-4t1f krb5kdc[20981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:31 dist-test-slave-4t1f krb5kdc[20981](info): set up 2 sockets
Jun 19 06:32:31 dist-test-slave-4t1f krb5kdc[20981](info): commencing operation
krb5kdc: starting...
W20260619 06:32:32.465339 389 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:32 dist-test-slave-4t1f krb5kdc[20981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:32.700137 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:33.014997 20994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:33.015262 20994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:33.015352 20994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:33.022368 20994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:33.022467 20994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:33.022513 20994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:33.022573 20994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:33.022616 20994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:33.031864 20994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:33.032966 20994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:33.034241 20994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:33.045611 20999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:33.045578 21000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:33.045538 21002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:33.046598 20994 server_base.cc:1034] running on GCE node
I20260619 06:32:33.047344 20994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:33.048588 20994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:33.049777 20994 hybrid_clock.cc:648] HybridClock initialized: now 1781850753049732 us; error 35 us; skew 500 ppm
I20260619 06:32:33.050107 20994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:33 dist-test-slave-4t1f krb5kdc[20981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850753, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:33.056020 20994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:33.057556 20994 webserver.cc:460] Webserver started at http://127.0.0.1:38039/ using document root <none> and password file <none>
I20260619 06:32:33.058192 20994 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:33.058290 20994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:33.058543 20994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:33.061496 20994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "ef2e85b375134819b8911d3be8165a75"
format_stamp: "Formatted at 2026-06-19 06:32:33 on dist-test-slave-4t1f"
server_key: "6af6445eda69bfc6b87495811643a3fa"
server_key_iv: "f8aba7dd689450030a56c085130a01d1"
server_key_version: "encryptionkey@0"
I20260619 06:32:33.062172 20994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "ef2e85b375134819b8911d3be8165a75"
format_stamp: "Formatted at 2026-06-19 06:32:33 on dist-test-slave-4t1f"
server_key: "6af6445eda69bfc6b87495811643a3fa"
server_key_iv: "f8aba7dd689450030a56c085130a01d1"
server_key_version: "encryptionkey@0"
I20260619 06:32:33.066991 20994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:32:33.070197 21009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:33.071478 20994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260619 06:32:33.071658 20994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "ef2e85b375134819b8911d3be8165a75"
format_stamp: "Formatted at 2026-06-19 06:32:33 on dist-test-slave-4t1f"
server_key: "6af6445eda69bfc6b87495811643a3fa"
server_key_iv: "f8aba7dd689450030a56c085130a01d1"
server_key_version: "encryptionkey@0"
I20260619 06:32:33.071851 20994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:33.077479 20994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:33.081300 20994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:33.081588 20994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:33.099519 20994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38757
I20260619 06:32:33.099547 21060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38757 every 8 connection(s)
I20260619 06:32:33.100927 20994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:33.101305 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 20994
I20260619 06:32:33.101497 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:33.102133 389 external_mini_cluster.cc:1428] Setting key 40dc6e74f04395ec925ebfab3c6989d0
I20260619 06:32:33.105149 21061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:33.109476 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:33.103794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47565 (local address 127.0.0.1:38757)
0619 06:32:33.104558 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:33.104569 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:33.104609 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:32:33.106316 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:33.106336 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:33.106639 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:33.106928 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:33.107940 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:33.107946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:33.107958 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:33.107976 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:33.108047 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:33.108066 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:33.108227 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:33.108326 (+ 99us) server_negotiation.cc:299] Negotiation successful
0619 06:32:33.108523 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":349,"threads_started":1}
I20260619 06:32:33.112920 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:33.116268 21061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75: Bootstrap starting.
I20260619 06:32:33.119457 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:33.117187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42193 (local address 127.0.0.1:38757)
0619 06:32:33.117385 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:33.117391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:33.117647 (+ 256us) server_negotiation.cc:407] Connection header received
0619 06:32:33.117807 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:33.117810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:33.117853 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:33.117924 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:33.118839 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:33.118843 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:33.118846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:33.118859 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:33.118891 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:33.118905 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:33.118989 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:33.119133 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 06:32:33.119171 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 06:32:33.119942 21061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:33.120888 21061 log.cc:824] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:33.123255 21061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75: No bootstrap required, opened a new log
I20260619 06:32:33.130158 21061 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2e85b375134819b8911d3be8165a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260619 06:32:33.130512 21061 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:33.130613 21061 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2e85b375134819b8911d3be8165a75, State: Initialized, Role: FOLLOWER
I20260619 06:32:33.131103 21061 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2e85b375134819b8911d3be8165a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260619 06:32:33.131289 21061 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:33.131379 21061 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:33.131491 21061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:33.133453 21061 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2e85b375134819b8911d3be8165a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260619 06:32:33.133826 21061 leader_election.cc:302] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef2e85b375134819b8911d3be8165a75; no voters:
I20260619 06:32:33.134320 21061 leader_election.cc:288] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:33.134456 21071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:33.135140 21071 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [term 1 LEADER]: Becoming Leader. State: Replica: ef2e85b375134819b8911d3be8165a75, State: Running, Role: LEADER
I20260619 06:32:33.135660 21071 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2e85b375134819b8911d3be8165a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } }
I20260619 06:32:33.136371 21061 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:33.138600 21072 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef2e85b375134819b8911d3be8165a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2e85b375134819b8911d3be8165a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } } }
I20260619 06:32:33.138677 21073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef2e85b375134819b8911d3be8165a75. Latest consensus state: current_term: 1 leader_uuid: "ef2e85b375134819b8911d3be8165a75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef2e85b375134819b8911d3be8165a75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38757 } } }
I20260619 06:32:33.138968 21072 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:33.139050 21073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:33.140997 21077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:33.146813 21077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:33.154829 21077 catalog_manager.cc:1347] Generated new cluster ID: b36d4b28bc0a42ec81ff8a707043dd56
I20260619 06:32:33.154935 21077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:33.169548 21077 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:33.169922 21077 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:33.178422 21077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef2e85b375134819b8911d3be8165a75: Generated new TSK 0
I20260619 06:32:33.179147 21077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:33.192624 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20994
W20260619 06:32:33.193643 21056 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47565
W20260619 06:32:33.194306 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 20994: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:33.194453 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 20994
2026-06-19T06:32:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1829 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:33 dist-test-slave-4t1f krb5kdc[21092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:33 dist-test-slave-4t1f krb5kdc[21092](info): set up 2 sockets
Jun 19 06:32:33 dist-test-slave-4t1f krb5kdc[21092](info): commencing operation
krb5kdc: starting...
W20260619 06:32:34.289479 389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:34 dist-test-slave-4t1f krb5kdc[21092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:34.518244 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:34.835237 21105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:34.835505 21105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:34.835675 21105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:34.842839 21105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:34.842962 21105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:34.843041 21105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:34.843130 21105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:34.843192 21105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:34.852735 21105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:34.853946 21105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:34.855140 21105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:34.866314 21110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:34.866374 21111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:34.866405 21113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:34.867156 21105 server_base.cc:1034] running on GCE node
I20260619 06:32:34.867882 21105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:34.869321 21105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:34.870538 21105 hybrid_clock.cc:648] HybridClock initialized: now 1781850754870501 us; error 50 us; skew 500 ppm
I20260619 06:32:34.870867 21105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:34 dist-test-slave-4t1f krb5kdc[21092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:34.873785 21105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:34.875181 21105 webserver.cc:460] Webserver started at http://127.0.0.1:33981/ using document root <none> and password file <none>
I20260619 06:32:34.875821 21105 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:34.875963 21105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:34.876253 21105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:34.879148 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "4edde6cd143a47fb81cf697c532f664c"
format_stamp: "Formatted at 2026-06-19 06:32:34 on dist-test-slave-4t1f"
server_key: "05bf456aa3309134cb2934f1f3657619"
server_key_iv: "67e7ebf98ce017df3235d00d4add9d97"
server_key_version: "encryptionkey@0"
I20260619 06:32:34.879874 21105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "4edde6cd143a47fb81cf697c532f664c"
format_stamp: "Formatted at 2026-06-19 06:32:34 on dist-test-slave-4t1f"
server_key: "05bf456aa3309134cb2934f1f3657619"
server_key_iv: "67e7ebf98ce017df3235d00d4add9d97"
server_key_version: "encryptionkey@0"
I20260619 06:32:34.884603 21105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 06:32:34.887754 21120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:34.888898 21105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260619 06:32:34.889153 21105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "4edde6cd143a47fb81cf697c532f664c"
format_stamp: "Formatted at 2026-06-19 06:32:34 on dist-test-slave-4t1f"
server_key: "05bf456aa3309134cb2934f1f3657619"
server_key_iv: "67e7ebf98ce017df3235d00d4add9d97"
server_key_version: "encryptionkey@0"
I20260619 06:32:34.889338 21105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:34.911108 21105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:34.914710 21105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:34.915009 21105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:34.932672 21105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35943
I20260619 06:32:34.932714 21171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35943 every 8 connection(s)
I20260619 06:32:34.934067 21105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:34.937508 21172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:34.940172 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21105
I20260619 06:32:34.940351 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:34.940855 389 external_mini_cluster.cc:1428] Setting key 2f956f40891abb1ee1031edbd94f5c33
I20260619 06:32:34.949329 21172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c: Bootstrap starting.
I20260619 06:32:34.952864 21172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:34.954195 21172 log.cc:824] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:32:34 dist-test-slave-4t1f krb5kdc[21092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:32:34.957280 21172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c: No bootstrap required, opened a new log
I20260619 06:32:34.963398 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:34.942574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35145 (local address 127.0.0.1:35943)
0619 06:32:34.943354 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:34.943364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:34.943402 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:32:34.944996 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:34.945016 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:34.945478 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:34.945782 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:34.947448 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:34.948727 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:34.950497 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:34.951132 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:34.951263 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:34.956512 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:34.956519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:34.956531 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:34.956574 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:34.959551 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:34.960397 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:34.960403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:34.960408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:34.960566 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:34.960953 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:34.960956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:34.960958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:34.961491 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:34.961711 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:34.962108 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 06:32:34.962409 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":377,"threads_started":1}
I20260619 06:32:34.966403 21172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4edde6cd143a47fb81cf697c532f664c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35943 } }
I20260619 06:32:34.966809 21172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:34.966961 21172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4edde6cd143a47fb81cf697c532f664c, State: Initialized, Role: FOLLOWER
I20260619 06:32:34.967079 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:34.967571 21172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4edde6cd143a47fb81cf697c532f664c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35943 } }
I20260619 06:32:34.967798 21172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:34.967937 21172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:34.968109 21172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:34.970511 21172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4edde6cd143a47fb81cf697c532f664c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35943 } }
I20260619 06:32:34.970996 21172 leader_election.cc:302] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4edde6cd143a47fb81cf697c532f664c; no voters:
I20260619 06:32:34.971668 21172 leader_election.cc:288] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:34.971761 21181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:34.974239 21172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:34.978792 21181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [term 1 LEADER]: Becoming Leader. State: Replica: 4edde6cd143a47fb81cf697c532f664c, State: Running, Role: LEADER
I20260619 06:32:34.979441 21181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4edde6cd143a47fb81cf697c532f664c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35943 } }
I20260619 06:32:34.986495 21184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4edde6cd143a47fb81cf697c532f664c. Latest consensus state: current_term: 1 leader_uuid: "4edde6cd143a47fb81cf697c532f664c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4edde6cd143a47fb81cf697c532f664c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35943 } } }
I20260619 06:32:34.986879 21184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:34.987383 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4edde6cd143a47fb81cf697c532f664c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4edde6cd143a47fb81cf697c532f664c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35943 } } }
I20260619 06:32:34.987665 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:34.988758 21196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:34.993327 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:34.971647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53349 (local address 127.0.0.1:35943)
0619 06:32:34.971856 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:34.971864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:34.977772 (+ 5908us) server_negotiation.cc:407] Connection header received
0619 06:32:34.977975 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:34.977979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:34.978036 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:34.978136 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:34.979586 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:34.980402 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:34.982107 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:34.982701 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:34.982770 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:34.986078 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:34.986105 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:34.986112 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:34.986153 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:34.990069 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:34.990930 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:34.990952 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:34.990958 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:34.991115 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:34.991544 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:34.991548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:34.991551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:34.992010 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:34.992266 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:34.992681 (+ 415us) server_negotiation.cc:299] Negotiation successful
0619 06:32:34.992945 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:32:34.993830 21196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:35.002632 21196 catalog_manager.cc:1347] Generated new cluster ID: 05e8754f1c5f4743b3fb1eed8f8213f2
I20260619 06:32:35.002765 21196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:35.030171 21196 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:35.031479 21196 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:35.044348 21196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4edde6cd143a47fb81cf697c532f664c: Generated new TSK 0
I20260619 06:32:35.045311 21196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:35.060001 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21105
I20260619 06:32:35.061805 21167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:35.155261 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21105
2026-06-19T06:32:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:35 dist-test-slave-4t1f krb5kdc[21204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:35 dist-test-slave-4t1f krb5kdc[21204](info): set up 2 sockets
Jun 19 06:32:35 dist-test-slave-4t1f krb5kdc[21204](info): commencing operation
krb5kdc: starting...
W20260619 06:32:36.249289 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:36 dist-test-slave-4t1f krb5kdc[21204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:36.480643 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:36.794677 21217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:36.794962 21217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:36.795075 21217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:36.802116 21217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:36.802248 21217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:36.802299 21217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:36.802371 21217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:36.802433 21217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:36.811599 21217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:36.812700 21217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:36.813961 21217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:36.825664 21222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:36.825585 21223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:36.825634 21225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:36.826320 21217 server_base.cc:1034] running on GCE node
I20260619 06:32:36.827041 21217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:36.828274 21217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:36.829502 21217 hybrid_clock.cc:648] HybridClock initialized: now 1781850756829464 us; error 49 us; skew 500 ppm
I20260619 06:32:36.829824 21217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:36 dist-test-slave-4t1f krb5kdc[21204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:36.832820 21217 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:36.834292 21217 webserver.cc:460] Webserver started at http://127.0.0.1:33495/ using document root <none> and password file <none>
I20260619 06:32:36.834868 21217 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:36.834964 21217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:36.835189 21217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:36.837867 21217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "a0b3daf2ba2e4e178573c08541a9f7c5"
format_stamp: "Formatted at 2026-06-19 06:32:36 on dist-test-slave-4t1f"
server_key: "d59ab17432eafc3fa00f571eb3e3a05b"
server_key_iv: "2a9711e6dbecd6fc0f36f3177e65e107"
server_key_version: "encryptionkey@0"
I20260619 06:32:36.838536 21217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "a0b3daf2ba2e4e178573c08541a9f7c5"
format_stamp: "Formatted at 2026-06-19 06:32:36 on dist-test-slave-4t1f"
server_key: "d59ab17432eafc3fa00f571eb3e3a05b"
server_key_iv: "2a9711e6dbecd6fc0f36f3177e65e107"
server_key_version: "encryptionkey@0"
I20260619 06:32:36.843375 21217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 06:32:36.846598 21232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:36.847808 21217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:32:36.847944 21217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "a0b3daf2ba2e4e178573c08541a9f7c5"
format_stamp: "Formatted at 2026-06-19 06:32:36 on dist-test-slave-4t1f"
server_key: "d59ab17432eafc3fa00f571eb3e3a05b"
server_key_iv: "2a9711e6dbecd6fc0f36f3177e65e107"
server_key_version: "encryptionkey@0"
I20260619 06:32:36.848109 21217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:36.879070 21217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:36.883080 21217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:36.883301 21217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:36.901341 21217 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43567
I20260619 06:32:36.901393 21283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43567 every 8 connection(s)
I20260619 06:32:36.902709 21217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:36.906466 21284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:36.912115 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21217
I20260619 06:32:36.912309 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:36.912855 389 external_mini_cluster.cc:1428] Setting key ffb09b5e18c0d6158a257d3499c98a71
I20260619 06:32:36.917913 21284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5: Bootstrap starting.
I20260619 06:32:36.920938 21284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:36.921950 21284 log.cc:824] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:36.924705 21284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5: No bootstrap required, opened a new log
I20260619 06:32:36.925809 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:36.914788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42867 (local address 127.0.0.1:43567)
0619 06:32:36.915641 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:36.915656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:36.915697 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:32:36.917433 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:36.917457 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:36.917841 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:36.918266 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:36.918814 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:36.920101 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:36.921529 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:36.922255 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:36.922442 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:36.923611 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:36.923619 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:36.923632 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:36.924142 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:36.924208 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:36.924231 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:36.924358 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:36.924582 (+ 224us) server_negotiation.cc:299] Negotiation successful
0619 06:32:36.924848 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":418,"threads_started":1}
I20260619 06:32:36.929334 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:36.932392 21284 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43567 } }
I20260619 06:32:36.932741 21284 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:36.932843 21284 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b3daf2ba2e4e178573c08541a9f7c5, State: Initialized, Role: FOLLOWER
I20260619 06:32:36.933544 21284 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43567 } }
I20260619 06:32:36.933866 21284 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:36.934012 21284 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:36.934224 21284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:36.937175 21284 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43567 } }
I20260619 06:32:36.937721 21284 leader_election.cc:302] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0b3daf2ba2e4e178573c08541a9f7c5; no voters:
I20260619 06:32:36.938455 21284 leader_election.cc:288] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:36.938596 21294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:36.940744 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:36.933530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37669 (local address 127.0.0.1:43567)
0619 06:32:36.933670 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:36.933674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:36.934448 (+ 774us) server_negotiation.cc:407] Connection header received
0619 06:32:36.934513 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:36.934516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:36.934570 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:36.934639 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:36.935311 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:36.936046 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:36.937920 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:36.938400 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:36.938558 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:36.939767 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:36.939772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:36.939776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:36.939798 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:36.939869 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:36.939888 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:36.939965 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:36.940431 (+ 466us) server_negotiation.cc:299] Negotiation successful
0619 06:32:36.940514 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:32:36.941053 21284 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:36.945560 21294 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [term 1 LEADER]: Becoming Leader. State: Replica: a0b3daf2ba2e4e178573c08541a9f7c5, State: Running, Role: LEADER
I20260619 06:32:36.946265 21294 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43567 } }
I20260619 06:32:36.949486 21295 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43567 } } }
I20260619 06:32:36.949575 21296 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0b3daf2ba2e4e178573c08541a9f7c5. Latest consensus state: current_term: 1 leader_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0b3daf2ba2e4e178573c08541a9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43567 } } }
I20260619 06:32:36.949988 21295 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:36.950083 21296 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:36.951382 21305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:36.956496 21305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:36.965229 21305 catalog_manager.cc:1347] Generated new cluster ID: 959d4629b271484ba87942eef8e9c1b1
I20260619 06:32:36.965338 21305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:36.983547 21305 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:36.985422 21305 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:37.000919 21305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0b3daf2ba2e4e178573c08541a9f7c5: Generated new TSK 0
I20260619 06:32:37.001950 21305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:37.008837 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21217
W20260619 06:32:37.010095 21279 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42867
W20260619 06:32:37.010828 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21217: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:37.010969 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21217
2026-06-19T06:32:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:37 dist-test-slave-4t1f krb5kdc[21315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:37 dist-test-slave-4t1f krb5kdc[21315](info): set up 2 sockets
Jun 19 06:32:37 dist-test-slave-4t1f krb5kdc[21315](info): commencing operation
krb5kdc: starting...
W20260619 06:32:38.129446 389 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:38 dist-test-slave-4t1f krb5kdc[21315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:38.362342 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:38.685491 21328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:38.685750 21328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:38.685856 21328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:38.693607 21328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:38.693720 21328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:38.693778 21328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:38.693845 21328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:38.693897 21328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:38.703646 21328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:38.704799 21328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:38.706111 21328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:38.717525 21333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:38.717607 21334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:38.717577 21336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:38.719156 21328 server_base.cc:1034] running on GCE node
I20260619 06:32:38.719856 21328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:38.721165 21328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:38.722384 21328 hybrid_clock.cc:648] HybridClock initialized: now 1781850758722341 us; error 38 us; skew 500 ppm
I20260619 06:32:38.722747 21328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:38 dist-test-slave-4t1f krb5kdc[21315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:38.729893 21328 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:38.731370 21328 webserver.cc:460] Webserver started at http://127.0.0.1:42759/ using document root <none> and password file <none>
I20260619 06:32:38.732035 21328 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:38.732182 21328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:38.732496 21328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:38.735527 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "7620228c6df94cc9a1c5f62f160213be"
format_stamp: "Formatted at 2026-06-19 06:32:38 on dist-test-slave-4t1f"
server_key: "8258bb0c15fc3961f9a7005db8bae2f8"
server_key_iv: "b1b3a97b64d51b3fade2715ca5f6d35d"
server_key_version: "encryptionkey@0"
I20260619 06:32:38.736310 21328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "7620228c6df94cc9a1c5f62f160213be"
format_stamp: "Formatted at 2026-06-19 06:32:38 on dist-test-slave-4t1f"
server_key: "8258bb0c15fc3961f9a7005db8bae2f8"
server_key_iv: "b1b3a97b64d51b3fade2715ca5f6d35d"
server_key_version: "encryptionkey@0"
I20260619 06:32:38.741541 21328 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 06:32:38.744879 21343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:38.746172 21328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:38.746369 21328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "7620228c6df94cc9a1c5f62f160213be"
format_stamp: "Formatted at 2026-06-19 06:32:38 on dist-test-slave-4t1f"
server_key: "8258bb0c15fc3961f9a7005db8bae2f8"
server_key_iv: "b1b3a97b64d51b3fade2715ca5f6d35d"
server_key_version: "encryptionkey@0"
I20260619 06:32:38.746568 21328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:38.752502 21328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:38.756340 21328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:38.756635 21328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:38.775709 21328 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46871
I20260619 06:32:38.775748 21394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46871 every 8 connection(s)
I20260619 06:32:38.777158 21328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:38.780524 21395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:38.783480 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21328
I20260619 06:32:38.783655 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:38.784204 389 external_mini_cluster.cc:1428] Setting key a87291263fd6134bd38d2a779290c8d2
I20260619 06:32:38.791872 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:38.786009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50747 (local address 127.0.0.1:46871)
0619 06:32:38.786804 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:38.786816 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:38.786858 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:32:38.788565 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:38.788586 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:38.788935 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:38.789353 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:38.790348 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:38.790354 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:38.790366 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:38.790396 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:38.790454 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:38.790478 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:38.790664 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:38.790772 (+ 108us) server_negotiation.cc:299] Negotiation successful
0619 06:32:38.790962 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":363,"threads_started":1}
I20260619 06:32:38.792418 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be: Bootstrap starting.
I20260619 06:32:38.795451 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:38.795442 21395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:38.796592 21395 log.cc:824] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:38.799721 21395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be: No bootstrap required, opened a new log
I20260619 06:32:38.802930 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:38.800326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38267 (local address 127.0.0.1:46871)
0619 06:32:38.800500 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:38.800505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:38.801009 (+ 504us) server_negotiation.cc:407] Connection header received
0619 06:32:38.801182 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:38.801186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:38.801241 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:38.801323 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:38.802229 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:38.802234 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:38.802237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:38.802251 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:38.802288 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:38.802301 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:38.802412 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:38.802534 (+ 122us) server_negotiation.cc:299] Negotiation successful
0619 06:32:38.802580 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 06:32:38.807669 21395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7620228c6df94cc9a1c5f62f160213be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260619 06:32:38.808012 21395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:38.808110 21395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7620228c6df94cc9a1c5f62f160213be, State: Initialized, Role: FOLLOWER
I20260619 06:32:38.808625 21395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7620228c6df94cc9a1c5f62f160213be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260619 06:32:38.808863 21395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:38.808995 21395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:38.809175 21395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:38.811303 21395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7620228c6df94cc9a1c5f62f160213be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260619 06:32:38.811767 21395 leader_election.cc:302] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7620228c6df94cc9a1c5f62f160213be; no voters:
I20260619 06:32:38.812281 21395 leader_election.cc:288] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:38.812415 21405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:38.813094 21405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [term 1 LEADER]: Becoming Leader. State: Replica: 7620228c6df94cc9a1c5f62f160213be, State: Running, Role: LEADER
I20260619 06:32:38.813611 21405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7620228c6df94cc9a1c5f62f160213be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } }
I20260619 06:32:38.814275 21395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:38.816346 21407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7620228c6df94cc9a1c5f62f160213be. Latest consensus state: current_term: 1 leader_uuid: "7620228c6df94cc9a1c5f62f160213be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7620228c6df94cc9a1c5f62f160213be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } } }
I20260619 06:32:38.816824 21407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:38.817233 21406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7620228c6df94cc9a1c5f62f160213be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7620228c6df94cc9a1c5f62f160213be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46871 } } }
I20260619 06:32:38.817571 21406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:38.819303 21410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:38.824364 21410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:38.832723 21410 catalog_manager.cc:1347] Generated new cluster ID: 04e37a50554947308ae3b6f4848ec564
I20260619 06:32:38.832821 21410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:38.864512 21410 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:38.864917 21410 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:38.871433 21410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7620228c6df94cc9a1c5f62f160213be: Generated new TSK 0
I20260619 06:32:38.872126 21410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:38.884166 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21328
W20260619 06:32:38.885156 21390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50747
W20260619 06:32:38.885787 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21328: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:38.885928 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21328
2026-06-19T06:32:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1875 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:38 dist-test-slave-4t1f krb5kdc[21426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:38 dist-test-slave-4t1f krb5kdc[21426](info): set up 2 sockets
Jun 19 06:32:38 dist-test-slave-4t1f krb5kdc[21426](info): commencing operation
krb5kdc: starting...
W20260619 06:32:39.973448 389 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:40 dist-test-slave-4t1f krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:40.234163 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42933
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:40.555575 21439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:40.555861 21439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:40.555956 21439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:40.563086 21439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:40.563225 21439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:40.563278 21439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:40.563331 21439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:40.563375 21439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:40.573361 21439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:40.574579 21439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:40.575932 21439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:40.587603 21445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:40.587684 21444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:40.587648 21447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:40.588629 21439 server_base.cc:1034] running on GCE node
I20260619 06:32:40.589444 21439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:40.590770 21439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:40.592006 21439 hybrid_clock.cc:648] HybridClock initialized: now 1781850760591950 us; error 44 us; skew 500 ppm
I20260619 06:32:40.592338 21439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:40 dist-test-slave-4t1f krb5kdc[21426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:40.595495 21439 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:40.596935 21439 webserver.cc:460] Webserver started at http://127.0.0.1:45035/ using document root <none> and password file <none>
I20260619 06:32:40.597651 21439 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:40.597790 21439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:40.598093 21439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:40.600965 21439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "2a4cacc3224549438acc9813d63326a0"
format_stamp: "Formatted at 2026-06-19 06:32:40 on dist-test-slave-4t1f"
server_key: "9225ea812c4c476a36cfe387f6de2824"
server_key_iv: "2a0e5ea4b3a3e4457bf19ea5e584f07d"
server_key_version: "encryptionkey@0"
I20260619 06:32:40.601740 21439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "2a4cacc3224549438acc9813d63326a0"
format_stamp: "Formatted at 2026-06-19 06:32:40 on dist-test-slave-4t1f"
server_key: "9225ea812c4c476a36cfe387f6de2824"
server_key_iv: "2a0e5ea4b3a3e4457bf19ea5e584f07d"
server_key_version: "encryptionkey@0"
I20260619 06:32:40.606603 21439 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 06:32:40.609838 21454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:40.611049 21439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:40.611220 21439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "2a4cacc3224549438acc9813d63326a0"
format_stamp: "Formatted at 2026-06-19 06:32:40 on dist-test-slave-4t1f"
server_key: "9225ea812c4c476a36cfe387f6de2824"
server_key_iv: "2a0e5ea4b3a3e4457bf19ea5e584f07d"
server_key_version: "encryptionkey@0"
I20260619 06:32:40.611420 21439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:40.625840 21439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:40.629927 21439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:40.630189 21439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:40.648049 21439 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42933
I20260619 06:32:40.648085 21505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42933 every 8 connection(s)
I20260619 06:32:40.649292 21439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:40.652829 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:40.656621 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21439
I20260619 06:32:40.656810 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:40.657394 389 external_mini_cluster.cc:1428] Setting key b80fc0ab06666d401ce5c9addcf4020e
I20260619 06:32:40.665414 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0: Bootstrap starting.
I20260619 06:32:40.669032 21506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:40.670353 21506 log.cc:824] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0: Log is configured to *not* fsync() on all Append() calls
Jun 19 06:32:40 dist-test-slave-4t1f krb5kdc[21426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:32:40.673216 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0: No bootstrap required, opened a new log
I20260619 06:32:40.681169 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:40.659105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:36363 (local address 127.0.0.1:42933)
0619 06:32:40.659793 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:40.659804 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:40.659840 (+ 36us) server_negotiation.cc:407] Connection header received
0619 06:32:40.661441 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:40.661461 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:40.661864 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:40.662175 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:40.663870 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:40.665183 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:40.666922 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:40.667564 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:40.667691 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:40.672642 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:40.672649 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:40.672662 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:40.672695 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:40.676716 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:40.677840 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:40.677846 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:40.677852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:40.678019 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:40.678502 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:40.678511 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:40.678514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:40.679013 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:40.679341 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:40.679789 (+ 448us) server_negotiation.cc:299] Negotiation successful
0619 06:32:40.680111 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":323,"threads_started":1}
I20260619 06:32:40.682421 21506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cacc3224549438acc9813d63326a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260619 06:32:40.682775 21506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:40.682891 21506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a4cacc3224549438acc9813d63326a0, State: Initialized, Role: FOLLOWER
I20260619 06:32:40.683499 21506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cacc3224549438acc9813d63326a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260619 06:32:40.683761 21506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:40.683893 21506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:40.684077 21506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:40.684970 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:40.687156 21506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cacc3224549438acc9813d63326a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260619 06:32:40.687754 21506 leader_election.cc:302] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a4cacc3224549438acc9813d63326a0; no voters:
I20260619 06:32:40.688426 21506 leader_election.cc:288] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:40.688558 21511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:40.690295 21511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [term 1 LEADER]: Becoming Leader. State: Replica: 2a4cacc3224549438acc9813d63326a0, State: Running, Role: LEADER
I20260619 06:32:40.690845 21511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cacc3224549438acc9813d63326a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } }
I20260619 06:32:40.691169 21506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:40.693877 21514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2a4cacc3224549438acc9813d63326a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cacc3224549438acc9813d63326a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } } }
I20260619 06:32:40.694221 21514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:40.693609 21516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a4cacc3224549438acc9813d63326a0. Latest consensus state: current_term: 1 leader_uuid: "2a4cacc3224549438acc9813d63326a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a4cacc3224549438acc9813d63326a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42933 } } }
I20260619 06:32:40.694811 21516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:40.705221 21521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:40.711534 21521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:40.714040 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:40.691779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:40187 (local address 127.0.0.1:42933)
0619 06:32:40.691972 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:40.691978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:40.692531 (+ 553us) server_negotiation.cc:407] Connection header received
0619 06:32:40.692853 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:40.692862 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:40.692992 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:40.693105 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:40.695612 (+ 2507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:40.697178 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:40.699501 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:40.700380 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:40.700494 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:40.704534 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:40.704543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:40.704554 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:40.704601 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:40.708552 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:40.709510 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:40.709515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:40.709519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:40.709680 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:40.710236 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:40.710243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:40.710248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:40.710742 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:40.711014 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:40.713263 (+ 2249us) server_negotiation.cc:299] Negotiation successful
0619 06:32:40.713508 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:32:40.720566 21521 catalog_manager.cc:1347] Generated new cluster ID: daea91a7f47342d38fdea5e55bddca7e
I20260619 06:32:40.720660 21521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:40.756906 21521 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:40.758791 21521 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:40.771904 21521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a4cacc3224549438acc9813d63326a0: Generated new TSK 0
I20260619 06:32:40.772732 21521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:40.783601 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21439
I20260619 06:32:40.784806 21501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:40.876837 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21439
W20260619 06:32:40.890523 21424 connection.cc:535] client connection to 127.0.0.1:42933 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42933): Connection reset by peer (error 104)
2026-06-19T06:32:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:40 dist-test-slave-4t1f krb5kdc[21539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:40 dist-test-slave-4t1f krb5kdc[21539](info): set up 2 sockets
Jun 19 06:32:40 dist-test-slave-4t1f krb5kdc[21539](info): commencing operation
krb5kdc: starting...
W20260619 06:32:41.974143 389 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:42 dist-test-slave-4t1f krb5kdc[21539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:42.210474 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:42.529110 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:42.529372 21552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:42.529553 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:42.536513 21552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:42.536607 21552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:42.536651 21552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:42.536732 21552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:42.536820 21552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:42.546283 21552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:42.547497 21552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:42.548821 21552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:42.560144 21558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:42.560138 21560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:42.560091 21557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:42.561182 21552 server_base.cc:1034] running on GCE node
I20260619 06:32:42.561892 21552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:42.563133 21552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:42.564354 21552 hybrid_clock.cc:648] HybridClock initialized: now 1781850762564312 us; error 60 us; skew 500 ppm
I20260619 06:32:42.564683 21552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:42 dist-test-slave-4t1f krb5kdc[21539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:42.568738 21552 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:42.570184 21552 webserver.cc:460] Webserver started at http://127.0.0.1:43417/ using document root <none> and password file <none>
I20260619 06:32:42.570815 21552 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:42.570953 21552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:42.571244 21552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:42.574100 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "f1b068033b4a4d6ab57ce738fa5c9766"
format_stamp: "Formatted at 2026-06-19 06:32:42 on dist-test-slave-4t1f"
server_key: "81446dbf1387ba3bc65ec7ac77af3574"
server_key_iv: "9c30bcd3bda2c33adb71368c9dc43263"
server_key_version: "encryptionkey@0"
I20260619 06:32:42.574820 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "f1b068033b4a4d6ab57ce738fa5c9766"
format_stamp: "Formatted at 2026-06-19 06:32:42 on dist-test-slave-4t1f"
server_key: "81446dbf1387ba3bc65ec7ac77af3574"
server_key_iv: "9c30bcd3bda2c33adb71368c9dc43263"
server_key_version: "encryptionkey@0"
I20260619 06:32:42.579578 21552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:32:42.582909 21567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:42.584154 21552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:32:42.584324 21552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "f1b068033b4a4d6ab57ce738fa5c9766"
format_stamp: "Formatted at 2026-06-19 06:32:42 on dist-test-slave-4t1f"
server_key: "81446dbf1387ba3bc65ec7ac77af3574"
server_key_iv: "9c30bcd3bda2c33adb71368c9dc43263"
server_key_version: "encryptionkey@0"
I20260619 06:32:42.584517 21552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:42.602274 21552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:42.605885 21552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:42.606137 21552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:42.623564 21552 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41245
I20260619 06:32:42.623605 21618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41245 every 8 connection(s)
I20260619 06:32:42.624806 21552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:42.628252 21619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:42.632884 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21552
I20260619 06:32:42.633121 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:42.633533 389 external_mini_cluster.cc:1428] Setting key ab6e479539ad9011ec74ed865d851f5e
I20260619 06:32:42.638799 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766: Bootstrap starting.
I20260619 06:32:42.642187 21619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:42.643290 21619 log.cc:824] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:42.646221 21619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766: No bootstrap required, opened a new log
I20260619 06:32:42.646488 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:42.635163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:51799 (local address 127.0.0.1:41245)
0619 06:32:42.635923 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:42.635934 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:42.635979 (+ 45us) server_negotiation.cc:407] Connection header received
0619 06:32:42.637724 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:42.637744 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:42.638099 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:42.638406 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:42.639000 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:42.640240 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:42.642135 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:42.642852 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:42.642993 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:42.644238 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:42.644250 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:42.644271 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:42.644909 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:42.644977 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:42.645000 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:42.645171 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:42.645405 (+ 234us) server_negotiation.cc:299] Negotiation successful
0619 06:32:42.645590 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":370,"threads_started":1}
I20260619 06:32:42.650218 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:42.656157 21619 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260619 06:32:42.656536 21619 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:42.656672 21619 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1b068033b4a4d6ab57ce738fa5c9766, State: Initialized, Role: FOLLOWER
I20260619 06:32:42.657271 21619 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260619 06:32:42.657562 21619 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:42.657692 21619 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:42.657895 21619 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:42.660406 21619 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260619 06:32:42.661034 21619 leader_election.cc:302] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1b068033b4a4d6ab57ce738fa5c9766; no voters:
I20260619 06:32:42.661912 21622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:42.655173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:59141 (local address 127.0.0.1:41245)
0619 06:32:42.655367 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:42.655374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:42.655790 (+ 416us) server_negotiation.cc:407] Connection header received
0619 06:32:42.655914 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:42.655919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:42.655991 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:42.656089 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:42.656545 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:42.657650 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:42.659009 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:42.659730 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:42.659849 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:42.660940 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:42.660946 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:42.660950 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:42.660969 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:42.661012 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:42.661037 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:42.661195 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:42.661389 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 06:32:42.661491 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:32:42.661959 21619 leader_election.cc:288] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:42.662062 21629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:42.663749 21629 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [term 1 LEADER]: Becoming Leader. State: Replica: f1b068033b4a4d6ab57ce738fa5c9766, State: Running, Role: LEADER
I20260619 06:32:42.664072 21619 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:42.664275 21629 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } }
I20260619 06:32:42.666991 21630 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } } }
I20260619 06:32:42.667912 21630 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:42.668187 21631 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1b068033b4a4d6ab57ce738fa5c9766. Latest consensus state: current_term: 1 leader_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1b068033b4a4d6ab57ce738fa5c9766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41245 } } }
I20260619 06:32:42.668512 21631 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:42.669895 21636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:42.676911 21636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:42.686683 21636 catalog_manager.cc:1347] Generated new cluster ID: a2c2fd35439e45e097209130393122e0
I20260619 06:32:42.686774 21636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:42.728094 21636 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:42.729406 21636 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:42.739637 21636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1b068033b4a4d6ab57ce738fa5c9766: Generated new TSK 0
I20260619 06:32:42.740435 21636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:42.749495 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21552
W20260619 06:32:42.750653 21614 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.172:51799
W20260619 06:32:42.751348 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21552: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:42.751493 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21552
W20260619 06:32:42.765031 21537 connection.cc:535] client connection to 127.0.0.1:41245 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41245): Connection reset by peer (error 104)
2026-06-19T06:32:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:42 dist-test-slave-4t1f krb5kdc[21650](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:42 dist-test-slave-4t1f krb5kdc[21650](info): set up 2 sockets
Jun 19 06:32:42 dist-test-slave-4t1f krb5kdc[21650](info): commencing operation
krb5kdc: starting...
W20260619 06:32:43.833614 389 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:43 dist-test-slave-4t1f krb5kdc[21650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:44.066286 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:44.390234 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:44.390583 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:44.390758 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:44.398063 21663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:44.398226 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:44.398346 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:44.398470 21663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:44.398552 21663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:44.408805 21663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:44.410228 21663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:44.411602 21663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:44.423746 21668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:44.423739 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:44.423786 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:44.424521 21663 server_base.cc:1034] running on GCE node
I20260619 06:32:44.425299 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:44.426882 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:44.428090 21663 hybrid_clock.cc:648] HybridClock initialized: now 1781850764428040 us; error 50 us; skew 500 ppm
I20260619 06:32:44.428440 21663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:44 dist-test-slave-4t1f krb5kdc[21650](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:44.435575 21663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:44.437600 21663 webserver.cc:460] Webserver started at http://127.0.0.1:33541/ using document root <none> and password file <none>
I20260619 06:32:44.438288 21663 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:44.438449 21663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:44.438771 21663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:44.441787 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "7275c9c36b174feb8478b103b1f32cdd"
format_stamp: "Formatted at 2026-06-19 06:32:44 on dist-test-slave-4t1f"
server_key: "fd1f84e253afbe9bddea36d3d27da2b0"
server_key_iv: "c6b6da5d7774f41540de4e361349b1b7"
server_key_version: "encryptionkey@0"
I20260619 06:32:44.442517 21663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "7275c9c36b174feb8478b103b1f32cdd"
format_stamp: "Formatted at 2026-06-19 06:32:44 on dist-test-slave-4t1f"
server_key: "fd1f84e253afbe9bddea36d3d27da2b0"
server_key_iv: "c6b6da5d7774f41540de4e361349b1b7"
server_key_version: "encryptionkey@0"
I20260619 06:32:44.447449 21663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:32:44.450847 21678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:44.452097 21663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 06:32:44.452280 21663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "7275c9c36b174feb8478b103b1f32cdd"
format_stamp: "Formatted at 2026-06-19 06:32:44 on dist-test-slave-4t1f"
server_key: "fd1f84e253afbe9bddea36d3d27da2b0"
server_key_iv: "c6b6da5d7774f41540de4e361349b1b7"
server_key_version: "encryptionkey@0"
I20260619 06:32:44.452473 21663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:44.458003 21663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:44.461761 21663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:44.462046 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:44.479780 21663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33983
I20260619 06:32:44.479817 21729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33983 every 8 connection(s)
I20260619 06:32:44.481268 21663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:44.481801 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21663
I20260619 06:32:44.481988 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:44.482564 389 external_mini_cluster.cc:1428] Setting key d735aec8798594b1f7c01cf9f857889a
I20260619 06:32:44.486115 21730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:44.490303 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:44.484405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:33607 (local address 127.0.0.1:33983)
0619 06:32:44.485199 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:44.485212 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:44.485253 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:32:44.486944 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:44.486966 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:44.487321 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:44.487702 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:44.488719 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:44.488727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:44.488740 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:44.488752 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:44.488824 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:44.488841 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:44.489122 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:44.489261 (+ 139us) server_negotiation.cc:299] Negotiation successful
0619 06:32:44.489434 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":320,"threads_started":1}
I20260619 06:32:44.493866 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:44.497810 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd: Bootstrap starting.
I20260619 06:32:44.501323 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:44.498973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:42691 (local address 127.0.0.1:33983)
0619 06:32:44.499162 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:44.499167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:44.499486 (+ 319us) server_negotiation.cc:407] Connection header received
0619 06:32:44.499657 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:44.499660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:44.499707 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:44.499783 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:44.500664 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:44.500667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:44.500671 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:44.500684 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:44.500723 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:44.500737 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:44.500818 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:44.500960 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 06:32:44.501004 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 06:32:44.501631 21730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:44.502902 21730 log.cc:824] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:44.505478 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd: No bootstrap required, opened a new log
I20260619 06:32:44.513163 21730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7275c9c36b174feb8478b103b1f32cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260619 06:32:44.513573 21730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:44.513677 21730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7275c9c36b174feb8478b103b1f32cdd, State: Initialized, Role: FOLLOWER
I20260619 06:32:44.514261 21730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7275c9c36b174feb8478b103b1f32cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260619 06:32:44.514494 21730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:44.514635 21730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:44.514806 21730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:44.517194 21730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7275c9c36b174feb8478b103b1f32cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260619 06:32:44.517683 21730 leader_election.cc:302] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7275c9c36b174feb8478b103b1f32cdd; no voters:
I20260619 06:32:44.518230 21730 leader_election.cc:288] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:44.518371 21740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:44.519040 21740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [term 1 LEADER]: Becoming Leader. State: Replica: 7275c9c36b174feb8478b103b1f32cdd, State: Running, Role: LEADER
I20260619 06:32:44.519562 21740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7275c9c36b174feb8478b103b1f32cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } }
I20260619 06:32:44.520300 21730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:44.522341 21742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7275c9c36b174feb8478b103b1f32cdd. Latest consensus state: current_term: 1 leader_uuid: "7275c9c36b174feb8478b103b1f32cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7275c9c36b174feb8478b103b1f32cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } } }
I20260619 06:32:44.522564 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7275c9c36b174feb8478b103b1f32cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7275c9c36b174feb8478b103b1f32cdd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33983 } } }
I20260619 06:32:44.522828 21742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:44.522850 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:44.524725 21748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:44.529676 21748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:44.538506 21748 catalog_manager.cc:1347] Generated new cluster ID: c8249ec5a520443cb11028213c8b090e
I20260619 06:32:44.538628 21748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:44.551857 21748 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:44.552259 21748 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:44.564909 21748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7275c9c36b174feb8478b103b1f32cdd: Generated new TSK 0
I20260619 06:32:44.565722 21748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:44.571004 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21663
W20260619 06:32:44.572202 21725 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.172:33607
W20260619 06:32:44.572834 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:44.572986 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21663
2026-06-19T06:32:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1824 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:44 dist-test-slave-4t1f krb5kdc[21761](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:44 dist-test-slave-4t1f krb5kdc[21761](info): set up 2 sockets
Jun 19 06:32:44 dist-test-slave-4t1f krb5kdc[21761](info): commencing operation
krb5kdc: starting...
W20260619 06:32:45.669526 389 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:45 dist-test-slave-4t1f krb5kdc[21761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:45.901997 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:46.225562 21774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:46.225827 21774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:46.225996 21774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:46.232970 21774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:46.233078 21774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:46.233143 21774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:46.233238 21774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:46.233301 21774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:46.246042 21774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:46.247413 21774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:46.248821 21774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:46.261130 21780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:46.261180 21782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:46.261492 21779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:46.261875 21774 server_base.cc:1034] running on GCE node
I20260619 06:32:46.262605 21774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:46.263923 21774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:46.265151 21774 hybrid_clock.cc:648] HybridClock initialized: now 1781850766265101 us; error 46 us; skew 500 ppm
I20260619 06:32:46.265473 21774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:46 dist-test-slave-4t1f krb5kdc[21761](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:46.268527 21774 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:46.269893 21774 webserver.cc:460] Webserver started at http://127.0.0.1:41523/ using document root <none> and password file <none>
I20260619 06:32:46.270509 21774 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:46.270642 21774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:46.270944 21774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:46.273802 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "b69db9e580584ba8b269286e1b85ad1b"
format_stamp: "Formatted at 2026-06-19 06:32:46 on dist-test-slave-4t1f"
server_key: "8c21aca39f27b50a2ed920ab4af24555"
server_key_iv: "6a387abbb6587e1fcb020056e85806b6"
server_key_version: "encryptionkey@0"
I20260619 06:32:46.274502 21774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "b69db9e580584ba8b269286e1b85ad1b"
format_stamp: "Formatted at 2026-06-19 06:32:46 on dist-test-slave-4t1f"
server_key: "8c21aca39f27b50a2ed920ab4af24555"
server_key_iv: "6a387abbb6587e1fcb020056e85806b6"
server_key_version: "encryptionkey@0"
I20260619 06:32:46.279434 21774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:32:46.282613 21789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:46.283797 21774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:32:46.283960 21774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "b69db9e580584ba8b269286e1b85ad1b"
format_stamp: "Formatted at 2026-06-19 06:32:46 on dist-test-slave-4t1f"
server_key: "8c21aca39f27b50a2ed920ab4af24555"
server_key_iv: "6a387abbb6587e1fcb020056e85806b6"
server_key_version: "encryptionkey@0"
I20260619 06:32:46.284133 21774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:46.296842 21774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:46.300930 21774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:46.301241 21774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:46.318485 21774 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46853
I20260619 06:32:46.318523 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46853 every 8 connection(s)
I20260619 06:32:46.319985 21774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:46.323021 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21774
I20260619 06:32:46.323184 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:46.323591 389 external_mini_cluster.cc:1428] Setting key a60b8689b50d9f2004f30a8160d86f7f
I20260619 06:32:46.323596 21841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:46.337699 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b: Bootstrap starting.
Jun 19 06:32:46 dist-test-slave-4t1f krb5kdc[21761](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 06:32:46.341265 21841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:46.342671 21841 log.cc:824] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:46.345995 21841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b: No bootstrap required, opened a new log
I20260619 06:32:46.347793 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:46.325346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:34259 (local address 127.0.0.1:46853)
0619 06:32:46.326107 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:46.326127 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:46.326177 (+ 50us) server_negotiation.cc:407] Connection header received
0619 06:32:46.328447 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:46.328475 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:46.329032 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:46.329481 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:46.330713 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:46.332689 (+ 1976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:46.334026 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:46.335115 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:46.335305 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:46.339751 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:46.339759 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:46.339772 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:46.339809 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:46.343621 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:46.344577 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:46.344583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:46.344589 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:46.344731 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:46.345265 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:46.345268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:46.345270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:46.345750 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:46.345992 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:46.346533 (+ 541us) server_negotiation.cc:299] Negotiation successful
0619 06:32:46.346823 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":283,"threads_started":1}
I20260619 06:32:46.351441 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:46.354665 21841 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69db9e580584ba8b269286e1b85ad1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46853 } }
I20260619 06:32:46.355113 21841 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:46.355224 21841 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69db9e580584ba8b269286e1b85ad1b, State: Initialized, Role: FOLLOWER
I20260619 06:32:46.355808 21841 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69db9e580584ba8b269286e1b85ad1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46853 } }
I20260619 06:32:46.356069 21841 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:46.356218 21841 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:46.356415 21841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:46.359598 21841 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69db9e580584ba8b269286e1b85ad1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46853 } }
I20260619 06:32:46.360140 21841 leader_election.cc:302] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69db9e580584ba8b269286e1b85ad1b; no voters:
I20260619 06:32:46.360755 21841 leader_election.cc:288] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:46.360867 21851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:46.362898 21841 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:46.365338 21851 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [term 1 LEADER]: Becoming Leader. State: Replica: b69db9e580584ba8b269286e1b85ad1b, State: Running, Role: LEADER
I20260619 06:32:46.366053 21851 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69db9e580584ba8b269286e1b85ad1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46853 } }
I20260619 06:32:46.376238 21844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:46.356072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:53247 (local address 127.0.0.1:46853)
0619 06:32:46.356366 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:46.356377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:46.356810 (+ 433us) server_negotiation.cc:407] Connection header received
0619 06:32:46.356991 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:46.356995 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:46.357055 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:46.357177 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:46.358970 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:46.359683 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:46.361500 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:46.362161 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:46.362268 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:46.365693 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:46.365698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:46.365701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:46.365729 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:46.370484 (+ 4755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:46.371835 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:46.371844 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:46.371850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:46.373320 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:46.374025 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:46.374033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:46.374038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:46.374457 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:46.374724 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:46.375555 (+ 831us) server_negotiation.cc:299] Negotiation successful
0619 06:32:46.375779 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 06:32:46.380306 21853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69db9e580584ba8b269286e1b85ad1b. Latest consensus state: current_term: 1 leader_uuid: "b69db9e580584ba8b269286e1b85ad1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69db9e580584ba8b269286e1b85ad1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46853 } } }
I20260619 06:32:46.380743 21853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:46.381393 21852 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69db9e580584ba8b269286e1b85ad1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69db9e580584ba8b269286e1b85ad1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46853 } } }
I20260619 06:32:46.381722 21852 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:46.394294 21861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:46.398954 21861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:46.407111 21861 catalog_manager.cc:1347] Generated new cluster ID: 3b186615474c437780b2253c85c2228f
I20260619 06:32:46.407207 21861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:46.444109 21861 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:46.445468 21861 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:46.461535 21861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b69db9e580584ba8b269286e1b85ad1b: Generated new TSK 0
I20260619 06:32:46.462458 21861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:46.471321 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21774
I20260619 06:32:46.473028 21836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:46.564674 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21774
W20260619 06:32:46.581418 21759 connection.cc:535] client connection to 127.0.0.1:46853 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46853): Connection reset by peer (error 104)
2026-06-19T06:32:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:46 dist-test-slave-4t1f krb5kdc[21874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:46 dist-test-slave-4t1f krb5kdc[21874](info): set up 2 sockets
Jun 19 06:32:46 dist-test-slave-4t1f krb5kdc[21874](info): commencing operation
krb5kdc: starting...
W20260619 06:32:47.661499 389 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:47 dist-test-slave-4t1f krb5kdc[21874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:47.912259 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:48.246858 21887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:48.247155 21887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:48.247326 21887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:48.255095 21887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:48.255208 21887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:48.255318 21887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:48.255431 21887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:48.255535 21887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:48.265659 21887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44003
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:48.266991 21887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:48.268342 21887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:48.280709 21892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:48.280650 21893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:48.280735 21895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:48.281092 21887 server_base.cc:1034] running on GCE node
I20260619 06:32:48.282178 21887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:48.283541 21887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:48.284741 21887 hybrid_clock.cc:648] HybridClock initialized: now 1781850768284693 us; error 45 us; skew 500 ppm
I20260619 06:32:48.285149 21887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:48 dist-test-slave-4t1f krb5kdc[21874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:48.288079 21887 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:48.289510 21887 webserver.cc:460] Webserver started at http://127.0.0.1:39667/ using document root <none> and password file <none>
I20260619 06:32:48.290203 21887 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:48.290344 21887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:48.290637 21887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:48.293531 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "718243a14cab4db9bd4e4fa313b1ac53"
format_stamp: "Formatted at 2026-06-19 06:32:48 on dist-test-slave-4t1f"
server_key: "42f038e7cfdb535f105f92cbad4d541c"
server_key_iv: "a94e9586a0abe9b7cd8bb5ccd6627257"
server_key_version: "encryptionkey@0"
I20260619 06:32:48.294253 21887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "718243a14cab4db9bd4e4fa313b1ac53"
format_stamp: "Formatted at 2026-06-19 06:32:48 on dist-test-slave-4t1f"
server_key: "42f038e7cfdb535f105f92cbad4d541c"
server_key_iv: "a94e9586a0abe9b7cd8bb5ccd6627257"
server_key_version: "encryptionkey@0"
I20260619 06:32:48.299113 21887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 06:32:48.302294 21902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:48.303499 21887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:48.303659 21887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "718243a14cab4db9bd4e4fa313b1ac53"
format_stamp: "Formatted at 2026-06-19 06:32:48 on dist-test-slave-4t1f"
server_key: "42f038e7cfdb535f105f92cbad4d541c"
server_key_iv: "a94e9586a0abe9b7cd8bb5ccd6627257"
server_key_version: "encryptionkey@0"
I20260619 06:32:48.303843 21887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:48.325630 21887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:48.329419 21887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:48.329712 21887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:48.347551 21887 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44003
I20260619 06:32:48.347589 21953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44003 every 8 connection(s)
I20260619 06:32:48.348810 21887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:48.352227 21954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:48.356711 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21887
I20260619 06:32:48.356887 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:48.357450 389 external_mini_cluster.cc:1428] Setting key 68da12cde5f179753a75b8e187677e36
I20260619 06:32:48.363385 21954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53: Bootstrap starting.
I20260619 06:32:48.366814 21954 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:48.368134 21954 log.cc:824] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:48.370923 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:48.359193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:53877 (local address 127.0.0.1:44003)
0619 06:32:48.359999 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:48.360012 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:48.360049 (+ 37us) server_negotiation.cc:407] Connection header received
0619 06:32:48.361970 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:48.361990 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:48.362343 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:48.362631 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:48.363185 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:48.364488 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:48.366365 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:48.367054 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:48.367187 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:48.368469 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:48.368475 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:48.368488 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:48.369284 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:48.369380 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:48.369413 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:48.369554 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:48.369876 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 06:32:48.370041 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":331,"threads_started":1}
I20260619 06:32:48.371129 21954 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53: No bootstrap required, opened a new log
I20260619 06:32:48.374447 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:48.380991 21954 raft_consensus.cc:348] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718243a14cab4db9bd4e4fa313b1ac53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260619 06:32:48.381455 21954 raft_consensus.cc:374] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:48.381574 21954 raft_consensus.cc:729] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 718243a14cab4db9bd4e4fa313b1ac53, State: Initialized, Role: FOLLOWER
I20260619 06:32:48.382252 21954 consensus_queue.cc:260] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718243a14cab4db9bd4e4fa313b1ac53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260619 06:32:48.382529 21954 raft_consensus.cc:388] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:48.382664 21954 raft_consensus.cc:482] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:48.382827 21954 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:48.385902 21954 raft_consensus.cc:504] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718243a14cab4db9bd4e4fa313b1ac53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260619 06:32:48.386379 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:48.379622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:42757 (local address 127.0.0.1:44003)
0619 06:32:48.379759 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:48.379763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:48.380116 (+ 353us) server_negotiation.cc:407] Connection header received
0619 06:32:48.380280 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:48.380283 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:48.380342 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:48.380416 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:48.380947 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:48.381876 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:48.383757 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:48.384227 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:48.384316 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:48.385592 (+ 1276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:48.385597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:48.385600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:48.385615 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:48.385650 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:48.385664 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:48.385743 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:48.385970 (+ 227us) server_negotiation.cc:299] Negotiation successful
0619 06:32:48.386011 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260619 06:32:48.386523 21954 leader_election.cc:302] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 718243a14cab4db9bd4e4fa313b1ac53; no voters:
I20260619 06:32:48.387255 21954 leader_election.cc:288] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:48.387377 21964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:48.389323 21964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [term 1 LEADER]: Becoming Leader. State: Replica: 718243a14cab4db9bd4e4fa313b1ac53, State: Running, Role: LEADER
I20260619 06:32:48.389948 21954 sys_catalog.cc:564] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:48.389952 21964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718243a14cab4db9bd4e4fa313b1ac53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } }
I20260619 06:32:48.393232 21965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "718243a14cab4db9bd4e4fa313b1ac53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718243a14cab4db9bd4e4fa313b1ac53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } } }
I20260619 06:32:48.393203 21966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 718243a14cab4db9bd4e4fa313b1ac53. Latest consensus state: current_term: 1 leader_uuid: "718243a14cab4db9bd4e4fa313b1ac53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "718243a14cab4db9bd4e4fa313b1ac53" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44003 } } }
I20260619 06:32:48.393677 21966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:48.393677 21965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:48.395596 21971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:48.401225 21971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:48.409893 21971 catalog_manager.cc:1347] Generated new cluster ID: ef002bb68417432ebf114dcbb890771c
I20260619 06:32:48.409991 21971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:48.423488 21971 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:48.425102 21971 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:48.431171 21971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 718243a14cab4db9bd4e4fa313b1ac53: Generated new TSK 0
I20260619 06:32:48.431963 21971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:48.439057 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21887
W20260619 06:32:48.440403 21949 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.172:53877
W20260619 06:32:48.441211 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21887: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:48.441401 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21887
W20260619 06:32:48.458950 21872 connection.cc:535] client connection to 127.0.0.1:44003 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44003): Connection reset by peer (error 104)
2026-06-19T06:32:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:48 dist-test-slave-4t1f krb5kdc[21985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:48 dist-test-slave-4t1f krb5kdc[21985](info): set up 2 sockets
Jun 19 06:32:48 dist-test-slave-4t1f krb5kdc[21985](info): commencing operation
krb5kdc: starting...
W20260619 06:32:49.537431 389 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:49 dist-test-slave-4t1f krb5kdc[21985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 06:32:49.778009 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37631
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:50.115243 21998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:50.115579 21998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:50.115736 21998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:50.122833 21998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:50.122978 21998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:50.123082 21998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:50.123190 21998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:50.123286 21998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:50.132659 21998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:50.133881 21998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:50.135136 21998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:50.146551 22004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:50.146591 22003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:50.146613 22006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:50.147611 21998 server_base.cc:1034] running on GCE node
I20260619 06:32:50.148291 21998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:50.149601 21998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:50.150802 21998 hybrid_clock.cc:648] HybridClock initialized: now 1781850770150768 us; error 46 us; skew 500 ppm
I20260619 06:32:50.151121 21998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:50 dist-test-slave-4t1f krb5kdc[21985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:50.154142 21998 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 06:32:50.155453 21998 webserver.cc:460] Webserver started at http://127.0.0.1:36645/ using document root <none> and password file <none>
I20260619 06:32:50.156075 21998 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:50.156212 21998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:50.156509 21998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:50.159385 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "30e7518b58f74b4da0a459f4e7255aae"
format_stamp: "Formatted at 2026-06-19 06:32:50 on dist-test-slave-4t1f"
server_key: "00db0e2bdd3306dd3ff09599a73da1dd"
server_key_iv: "d867a0f7e25545edf5b22fd8b803f230"
server_key_version: "encryptionkey@0"
I20260619 06:32:50.160100 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "30e7518b58f74b4da0a459f4e7255aae"
format_stamp: "Formatted at 2026-06-19 06:32:50 on dist-test-slave-4t1f"
server_key: "00db0e2bdd3306dd3ff09599a73da1dd"
server_key_iv: "d867a0f7e25545edf5b22fd8b803f230"
server_key_version: "encryptionkey@0"
I20260619 06:32:50.164958 21998 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260619 06:32:50.168032 22013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:50.169277 21998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:32:50.169442 21998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "30e7518b58f74b4da0a459f4e7255aae"
format_stamp: "Formatted at 2026-06-19 06:32:50 on dist-test-slave-4t1f"
server_key: "00db0e2bdd3306dd3ff09599a73da1dd"
server_key_iv: "d867a0f7e25545edf5b22fd8b803f230"
server_key_version: "encryptionkey@0"
I20260619 06:32:50.169625 21998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:50.174981 21998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:50.179246 21998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:50.179550 21998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:50.196969 21998 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37631
I20260619 06:32:50.197012 22064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37631 every 8 connection(s)
I20260619 06:32:50.198324 21998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:50.201364 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 21998
I20260619 06:32:50.201546 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:50.201857 22065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:50.202059 389 external_mini_cluster.cc:1428] Setting key 2af12401f7192cf715dabfb38d178bf7
I20260619 06:32:50.209633 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:50.203721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:54109 (local address 127.0.0.1:37631)
0619 06:32:50.204530 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:50.204552 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:50.204594 (+ 42us) server_negotiation.cc:407] Connection header received
0619 06:32:50.206290 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:50.206311 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:50.206618 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:50.206964 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:50.208030 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:50.208038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:50.208062 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:50.208081 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:50.208146 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:50.208170 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:50.208350 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:50.208447 (+ 97us) server_negotiation.cc:299] Negotiation successful
0619 06:32:50.208682 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":369,"threads_started":1}
I20260619 06:32:50.213099 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:50.215067 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae: Bootstrap starting.
I20260619 06:32:50.219076 22065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:50.220407 22065 log.cc:824] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:50.220530 22068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:50.218243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.172:52767 (local address 127.0.0.1:37631)
0619 06:32:50.218439 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:50.218445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:50.218695 (+ 250us) server_negotiation.cc:407] Connection header received
0619 06:32:50.218875 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:50.218878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:50.218922 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:50.219001 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:50.219877 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:50.219880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:50.219883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:50.219897 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:50.219932 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:50.219946 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:50.220030 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:50.220160 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 06:32:50.220207 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 06:32:50.223259 22065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae: No bootstrap required, opened a new log
I20260619 06:32:50.230547 22065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e7518b58f74b4da0a459f4e7255aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260619 06:32:50.230880 22065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:50.230988 22065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30e7518b58f74b4da0a459f4e7255aae, State: Initialized, Role: FOLLOWER
I20260619 06:32:50.231570 22065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e7518b58f74b4da0a459f4e7255aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260619 06:32:50.231802 22065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:50.231925 22065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:50.232085 22065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:50.234251 22065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e7518b58f74b4da0a459f4e7255aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260619 06:32:50.234709 22065 leader_election.cc:302] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30e7518b58f74b4da0a459f4e7255aae; no voters:
I20260619 06:32:50.235257 22065 leader_election.cc:288] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:50.235400 22075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:50.236059 22075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [term 1 LEADER]: Becoming Leader. State: Replica: 30e7518b58f74b4da0a459f4e7255aae, State: Running, Role: LEADER
I20260619 06:32:50.236548 22075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e7518b58f74b4da0a459f4e7255aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } }
I20260619 06:32:50.237505 22065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:50.239507 22077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30e7518b58f74b4da0a459f4e7255aae. Latest consensus state: current_term: 1 leader_uuid: "30e7518b58f74b4da0a459f4e7255aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e7518b58f74b4da0a459f4e7255aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } } }
I20260619 06:32:50.239959 22077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:50.240918 22076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30e7518b58f74b4da0a459f4e7255aae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30e7518b58f74b4da0a459f4e7255aae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37631 } } }
I20260619 06:32:50.241317 22076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:50.241626 22083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:50.246867 22083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:50.255640 22083 catalog_manager.cc:1347] Generated new cluster ID: cbb50971f48848dea2c51929d389fd33
I20260619 06:32:50.255761 22083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:50.278646 22083 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:50.279062 22083 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:50.285742 22083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30e7518b58f74b4da0a459f4e7255aae: Generated new TSK 0
I20260619 06:32:50.286729 22083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:50.295370 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21998
W20260619 06:32:50.296418 22060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.172:54109
W20260619 06:32:50.297122 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 21998: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:50.297263 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 21998
2026-06-19T06:32:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1854 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:50 dist-test-slave-4t1f krb5kdc[22096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:50 dist-test-slave-4t1f krb5kdc[22096](info): set up 2 sockets
Jun 19 06:32:50 dist-test-slave-4t1f krb5kdc[22096](info): commencing operation
krb5kdc: starting...
W20260619 06:32:51.389567 389 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.007s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:51 dist-test-slave-4t1f krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:51.648150 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36061
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:34701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:51.970888 22109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:51.971204 22109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:51.971295 22109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:51.978595 22109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:51.978725 22109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:51.978780 22109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:51.978835 22109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:51.978881 22109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:51.988814 22109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:34701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36061
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:51.990183 22109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:51.991420 22109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:52.002941 22114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:52.003232 22117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:52.002969 22115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:52.004046 22109 server_base.cc:1034] running on GCE node
I20260619 06:32:52.004828 22109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:52.006341 22109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:52.007575 22109 hybrid_clock.cc:648] HybridClock initialized: now 1781850772007534 us; error 48 us; skew 500 ppm
I20260619 06:32:52.007928 22109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:52 dist-test-slave-4t1f krb5kdc[22096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:52.011088 22109 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:52.012485 22109 webserver.cc:460] Webserver started at http://127.0.97.126:38751/ using document root <none> and password file <none>
I20260619 06:32:52.013258 22109 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:52.013403 22109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:52.013734 22109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:52.016738 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "558310f2f0fd4371910bd04be83356ec"
format_stamp: "Formatted at 2026-06-19 06:32:52 on dist-test-slave-4t1f"
server_key: "d5fad7cfbeca16a8b47b8063ad6fc6e9"
server_key_iv: "7aaa1457bbbcd63986358657fd8cd53b"
server_key_version: "encryptionkey@0"
I20260619 06:32:52.017560 22109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "558310f2f0fd4371910bd04be83356ec"
format_stamp: "Formatted at 2026-06-19 06:32:52 on dist-test-slave-4t1f"
server_key: "d5fad7cfbeca16a8b47b8063ad6fc6e9"
server_key_iv: "7aaa1457bbbcd63986358657fd8cd53b"
server_key_version: "encryptionkey@0"
I20260619 06:32:52.022658 22109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 06:32:52.026069 22124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:52.027374 22109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 06:32:52.027570 22109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "558310f2f0fd4371910bd04be83356ec"
format_stamp: "Formatted at 2026-06-19 06:32:52 on dist-test-slave-4t1f"
server_key: "d5fad7cfbeca16a8b47b8063ad6fc6e9"
server_key_iv: "7aaa1457bbbcd63986358657fd8cd53b"
server_key_version: "encryptionkey@0"
I20260619 06:32:52.027770 22109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:52.044876 22109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:52.049407 22109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:52.049698 22109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:52.068701 22109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36061
I20260619 06:32:52.068725 22175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36061 every 8 connection(s)
I20260619 06:32:52.070200 22109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:52.072010 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22109
I20260619 06:32:52.072181 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:52.072731 389 external_mini_cluster.cc:1428] Setting key ffd0fde594e03c829e51aa498745ecc3
I20260619 06:32:52.074852 22176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:52.092269 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec: Bootstrap starting.
Jun 19 06:32:52 dist-test-slave-4t1f krb5kdc[22096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:52.095954 22176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:52.097123 22176 log.cc:824] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:52.100020 22176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec: No bootstrap required, opened a new log
I20260619 06:32:52.103082 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:52.074549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35289 (local address 127.0.97.126:36061)
0619 06:32:52.081043 (+ 6494us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:52.081095 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:52.081160 (+ 65us) server_negotiation.cc:407] Connection header received
0619 06:32:52.083942 (+ 2782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:52.083976 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:52.084616 (+ 640us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:52.085117 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:52.086763 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:52.088006 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:52.089906 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:52.090587 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:52.090721 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:52.095560 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:52.095569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:52.095584 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:52.095638 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:52.098938 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:52.099759 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:52.099765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:52.099771 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:52.099935 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:52.100423 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:52.100426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:52.100428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:52.100924 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:52.101284 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:52.101787 (+ 503us) server_negotiation.cc:299] Negotiation successful
0619 06:32:52.102092 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6153,"thread_start_us":6489,"threads_started":1}
I20260619 06:32:52.106961 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:52.108428 22176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558310f2f0fd4371910bd04be83356ec" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36061 } }
I20260619 06:32:52.108837 22176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:52.108942 22176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 558310f2f0fd4371910bd04be83356ec, State: Initialized, Role: FOLLOWER
I20260619 06:32:52.109555 22176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558310f2f0fd4371910bd04be83356ec" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36061 } }
I20260619 06:32:52.109830 22176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:52.109969 22176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:52.110154 22176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:52.112622 22176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558310f2f0fd4371910bd04be83356ec" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36061 } }
I20260619 06:32:52.113171 22176 leader_election.cc:302] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 558310f2f0fd4371910bd04be83356ec; no voters:
I20260619 06:32:52.113828 22176 leader_election.cc:288] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:52.113976 22186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:52.115207 22186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [term 1 LEADER]: Becoming Leader. State: Replica: 558310f2f0fd4371910bd04be83356ec, State: Running, Role: LEADER
I20260619 06:32:52.115722 22186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558310f2f0fd4371910bd04be83356ec" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36061 } }
I20260619 06:32:52.116016 22176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:52.118471 22188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 558310f2f0fd4371910bd04be83356ec. Latest consensus state: current_term: 1 leader_uuid: "558310f2f0fd4371910bd04be83356ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558310f2f0fd4371910bd04be83356ec" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36061 } } }
I20260619 06:32:52.118482 22187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "558310f2f0fd4371910bd04be83356ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "558310f2f0fd4371910bd04be83356ec" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36061 } } }
I20260619 06:32:52.118899 22188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:52.118899 22187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:52.119956 22191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:52.132150 22179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:52.111389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41127 (local address 127.0.97.126:36061)
0619 06:32:52.111568 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:52.111574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:52.112005 (+ 431us) server_negotiation.cc:407] Connection header received
0619 06:32:52.112123 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:52.112129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:52.112197 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:52.112300 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:52.114007 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:52.115242 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:52.117043 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:52.118021 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:52.118173 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:52.122290 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:52.122299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:52.122308 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:52.122357 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:52.127051 (+ 4694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:52.128295 (+ 1244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:52.128304 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:52.128310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:52.128478 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:52.129165 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:52.129173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:52.129179 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:52.129659 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:52.129935 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:52.131347 (+ 1412us) server_negotiation.cc:299] Negotiation successful
0619 06:32:52.131582 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:32:52.137754 22191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:52.147533 22191 catalog_manager.cc:1347] Generated new cluster ID: 7427b2769bfc405daa588dd96d0272b6
I20260619 06:32:52.147687 22191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:52.166352 22191 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:52.168124 22191 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:52.178084 22191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 558310f2f0fd4371910bd04be83356ec: Generated new TSK 0
I20260619 06:32:52.179040 22191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:52.189983 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22109
I20260619 06:32:52.191124 22171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:52.282869 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22109
W20260619 06:32:52.300000 22094 connection.cc:535] client connection to 127.0.97.126:36061 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.126:36061): Connection reset by peer (error 104)
2026-06-19T06:32:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1986 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:52 dist-test-slave-4t1f krb5kdc[22208](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:52 dist-test-slave-4t1f krb5kdc[22208](info): set up 2 sockets
Jun 19 06:32:52 dist-test-slave-4t1f krb5kdc[22208](info): commencing operation
krb5kdc: starting...
W20260619 06:32:53.389528 389 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:53 dist-test-slave-4t1f krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:53.634465 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35259
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:53.956559 22221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:53.956810 22221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:53.956964 22221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:53.963935 22221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:53.964023 22221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:53.964131 22221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:53.964231 22221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:53.964304 22221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:53.973670 22221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:35259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:53.974870 22221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:53.976090 22221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:53.987229 22226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:53.987255 22227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:53.987164 22229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:53.987998 22221 server_base.cc:1034] running on GCE node
I20260619 06:32:53.988770 22221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:53.990072 22221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:53.991259 22221 hybrid_clock.cc:648] HybridClock initialized: now 1781850773991224 us; error 34 us; skew 500 ppm
I20260619 06:32:53.991571 22221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:53 dist-test-slave-4t1f krb5kdc[22208](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850773, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:53.994628 22221 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:53.996001 22221 webserver.cc:460] Webserver started at http://127.0.97.126:39839/ using document root <none> and password file <none>
I20260619 06:32:53.996610 22221 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:53.996768 22221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:53.997128 22221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:53.999838 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "9f59522ca777411386313ca081e074cf"
format_stamp: "Formatted at 2026-06-19 06:32:53 on dist-test-slave-4t1f"
server_key: "944e77679cff9459d760065388e4da79"
server_key_iv: "63d94bfadb0d5a942af2470adce46cc4"
server_key_version: "encryptionkey@0"
I20260619 06:32:54.000553 22221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "9f59522ca777411386313ca081e074cf"
format_stamp: "Formatted at 2026-06-19 06:32:53 on dist-test-slave-4t1f"
server_key: "944e77679cff9459d760065388e4da79"
server_key_iv: "63d94bfadb0d5a942af2470adce46cc4"
server_key_version: "encryptionkey@0"
I20260619 06:32:54.005376 22221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260619 06:32:54.008615 22236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:54.009859 22221 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:54.010049 22221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "9f59522ca777411386313ca081e074cf"
format_stamp: "Formatted at 2026-06-19 06:32:53 on dist-test-slave-4t1f"
server_key: "944e77679cff9459d760065388e4da79"
server_key_iv: "63d94bfadb0d5a942af2470adce46cc4"
server_key_version: "encryptionkey@0"
I20260619 06:32:54.010228 22221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:54.024566 22221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:54.028147 22221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:54.028434 22221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:54.046275 22221 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:35259
I20260619 06:32:54.046342 22287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:35259 every 8 connection(s)
I20260619 06:32:54.047582 22221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:54.051132 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:54.056808 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22221
I20260619 06:32:54.057008 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:54.057461 389 external_mini_cluster.cc:1428] Setting key be645d4db6d5be73fd4a2c79a2cef053
I20260619 06:32:54.061656 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf: Bootstrap starting.
I20260619 06:32:54.065512 22288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:54.066752 22288 log.cc:824] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:54.069969 22288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf: No bootstrap required, opened a new log
I20260619 06:32:54.071461 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:54.059256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48783 (local address 127.0.97.126:35259)
0619 06:32:54.060060 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:54.060070 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:54.060113 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:32:54.061862 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:54.061883 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:54.062231 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:54.062527 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:54.063169 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:54.064942 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:54.066980 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:54.068004 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:54.068177 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:54.069381 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:54.069389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:54.069402 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:54.069893 (+ 491us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:54.069964 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:54.069987 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:54.070165 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:54.070444 (+ 279us) server_negotiation.cc:299] Negotiation successful
0619 06:32:54.070594 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":406,"threads_started":1}
I20260619 06:32:54.075661 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:54.080495 22288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59522ca777411386313ca081e074cf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35259 } }
I20260619 06:32:54.080950 22288 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:54.081110 22288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f59522ca777411386313ca081e074cf, State: Initialized, Role: FOLLOWER
I20260619 06:32:54.081990 22288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59522ca777411386313ca081e074cf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35259 } }
I20260619 06:32:54.082253 22288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:54.082407 22288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:54.082609 22288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:54.084826 22288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59522ca777411386313ca081e074cf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35259 } }
I20260619 06:32:54.085391 22288 leader_election.cc:302] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f59522ca777411386313ca081e074cf; no voters:
I20260619 06:32:54.086051 22288 leader_election.cc:288] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:54.086153 22298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:54.086853 22298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [term 1 LEADER]: Becoming Leader. State: Replica: 9f59522ca777411386313ca081e074cf, State: Running, Role: LEADER
I20260619 06:32:54.087445 22298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59522ca777411386313ca081e074cf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35259 } }
I20260619 06:32:54.087945 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:54.081366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56255 (local address 127.0.97.126:35259)
0619 06:32:54.081564 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:54.081570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:54.081964 (+ 394us) server_negotiation.cc:407] Connection header received
0619 06:32:54.082179 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:54.082186 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:54.082272 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:54.082382 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:54.082900 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:54.083699 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:54.084963 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:54.085518 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:54.085654 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:54.087060 (+ 1406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:54.087066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:54.087070 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:54.087088 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:54.087131 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:54.087149 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:54.087239 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:54.087636 (+ 397us) server_negotiation.cc:299] Negotiation successful
0619 06:32:54.087676 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:32:54.088335 22288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:54.091614 22300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f59522ca777411386313ca081e074cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59522ca777411386313ca081e074cf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35259 } } }
I20260619 06:32:54.091639 22299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f59522ca777411386313ca081e074cf. Latest consensus state: current_term: 1 leader_uuid: "9f59522ca777411386313ca081e074cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59522ca777411386313ca081e074cf" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 35259 } } }
I20260619 06:32:54.092053 22300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:54.092061 22299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:54.092976 22308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:54.098670 22308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:54.108155 22308 catalog_manager.cc:1347] Generated new cluster ID: bc957502aa0d451c9487ea13935697c4
I20260619 06:32:54.108270 22308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:54.122690 22308 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:54.123905 22308 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:54.141362 22308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f59522ca777411386313ca081e074cf: Generated new TSK 0
I20260619 06:32:54.142204 22308 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:54.152460 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22221
W20260619 06:32:54.153513 22283 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48783
W20260619 06:32:54.154296 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22221: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:54.154428 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22221
W20260619 06:32:54.170773 22206 connection.cc:535] client connection to 127.0.97.126:35259 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.126:35259): Connection reset by peer (error 104)
2026-06-19T06:32:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1870 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:54 dist-test-slave-4t1f krb5kdc[22319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:54 dist-test-slave-4t1f krb5kdc[22319](info): set up 2 sockets
Jun 19 06:32:54 dist-test-slave-4t1f krb5kdc[22319](info): commencing operation
krb5kdc: starting...
W20260619 06:32:55.249639 389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:55 dist-test-slave-4t1f krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850775, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:55.493819 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36929
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:42591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:55.819109 22332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:55.819453 22332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:55.819622 22332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:55.826813 22332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:55.826978 22332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:55.827081 22332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:55.827205 22332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:55.827278 22332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:55.837049 22332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:42591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:55.838450 22332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:55.839777 22332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:55.851608 22338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:55.851708 22337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:55.851698 22340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:55.852443 22332 server_base.cc:1034] running on GCE node
I20260619 06:32:55.853193 22332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:55.854480 22332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:55.855681 22332 hybrid_clock.cc:648] HybridClock initialized: now 1781850775855646 us; error 38 us; skew 500 ppm
I20260619 06:32:55.855968 22332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:55 dist-test-slave-4t1f krb5kdc[22319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850775, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:55.862030 22332 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:55.863502 22332 webserver.cc:460] Webserver started at http://127.0.97.126:34449/ using document root <none> and password file <none>
I20260619 06:32:55.864149 22332 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:55.864295 22332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:55.864578 22332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:55.867409 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "832b5e74a7e942c698c320b62071f90e"
format_stamp: "Formatted at 2026-06-19 06:32:55 on dist-test-slave-4t1f"
server_key: "3f67ad788caea70a67065d98539182a3"
server_key_iv: "021938f68fc1799a6aeef9a5ee301c18"
server_key_version: "encryptionkey@0"
I20260619 06:32:55.868145 22332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "832b5e74a7e942c698c320b62071f90e"
format_stamp: "Formatted at 2026-06-19 06:32:55 on dist-test-slave-4t1f"
server_key: "3f67ad788caea70a67065d98539182a3"
server_key_iv: "021938f68fc1799a6aeef9a5ee301c18"
server_key_version: "encryptionkey@0"
I20260619 06:32:55.873243 22332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260619 06:32:55.876785 22347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:55.878058 22332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:32:55.878242 22332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "832b5e74a7e942c698c320b62071f90e"
format_stamp: "Formatted at 2026-06-19 06:32:55 on dist-test-slave-4t1f"
server_key: "3f67ad788caea70a67065d98539182a3"
server_key_iv: "021938f68fc1799a6aeef9a5ee301c18"
server_key_version: "encryptionkey@0"
I20260619 06:32:55.878451 22332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:55.883956 22332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:55.887740 22332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:55.888015 22332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:55.906016 22332 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36929
I20260619 06:32:55.906090 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36929 every 8 connection(s)
I20260619 06:32:55.907382 22332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:55.911301 22399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:55.916843 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22332
I20260619 06:32:55.917026 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:55.917627 389 external_mini_cluster.cc:1428] Setting key 154d8752a6848d204d2c77b279bba889
I20260619 06:32:55.923400 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e: Bootstrap starting.
I20260619 06:32:55.925525 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:55.919419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33899 (local address 127.0.97.126:36929)
0619 06:32:55.920321 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:55.920332 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:55.920370 (+ 38us) server_negotiation.cc:407] Connection header received
0619 06:32:55.922160 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:55.922181 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:55.922493 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:55.922785 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:55.923880 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:55.923886 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:55.923898 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:55.923911 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:55.923973 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:55.923998 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:55.924162 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:55.924342 (+ 180us) server_negotiation.cc:299] Negotiation successful
0619 06:32:55.924535 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":492,"threads_started":1}
I20260619 06:32:55.927066 22399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:55.928269 22399 log.cc:824] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:55.928959 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:55.931102 22399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e: No bootstrap required, opened a new log
I20260619 06:32:55.935657 22402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:55.933163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35205 (local address 127.0.97.126:36929)
0619 06:32:55.933354 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:55.933361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:55.933770 (+ 409us) server_negotiation.cc:407] Connection header received
0619 06:32:55.933892 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:55.933898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:55.933971 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:55.934088 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:55.934968 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:55.934974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:55.934979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:55.934999 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:55.935049 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:55.935076 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:55.935221 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:55.935285 (+ 64us) server_negotiation.cc:299] Negotiation successful
0619 06:32:55.935343 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 06:32:55.940212 22399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832b5e74a7e942c698c320b62071f90e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36929 } }
I20260619 06:32:55.940659 22399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:55.940795 22399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832b5e74a7e942c698c320b62071f90e, State: Initialized, Role: FOLLOWER
I20260619 06:32:55.941583 22399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832b5e74a7e942c698c320b62071f90e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36929 } }
I20260619 06:32:55.941813 22399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:55.941924 22399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:55.942066 22399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:55.944283 22399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832b5e74a7e942c698c320b62071f90e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36929 } }
I20260619 06:32:55.944730 22399 leader_election.cc:302] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 832b5e74a7e942c698c320b62071f90e; no voters:
I20260619 06:32:55.945318 22399 leader_election.cc:288] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:55.945439 22409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:55.946273 22409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [term 1 LEADER]: Becoming Leader. State: Replica: 832b5e74a7e942c698c320b62071f90e, State: Running, Role: LEADER
I20260619 06:32:55.946833 22409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832b5e74a7e942c698c320b62071f90e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36929 } }
I20260619 06:32:55.947775 22399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:55.950140 22411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 832b5e74a7e942c698c320b62071f90e. Latest consensus state: current_term: 1 leader_uuid: "832b5e74a7e942c698c320b62071f90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832b5e74a7e942c698c320b62071f90e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36929 } } }
I20260619 06:32:55.950186 22410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "832b5e74a7e942c698c320b62071f90e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "832b5e74a7e942c698c320b62071f90e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36929 } } }
I20260619 06:32:55.950533 22411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:55.950618 22410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:55.951995 22415 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:55.956871 22415 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:55.966096 22415 catalog_manager.cc:1347] Generated new cluster ID: 4e124b8626524448b626f3d233831e8e
I20260619 06:32:55.966187 22415 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:55.992075 22415 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:55.992508 22415 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:56.002179 22415 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 832b5e74a7e942c698c320b62071f90e: Generated new TSK 0
I20260619 06:32:56.002936 22415 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:56.008775 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22332
W20260619 06:32:56.010078 22394 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33899
W20260619 06:32:56.010792 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22332: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:56.010962 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22332
2026-06-19T06:32:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1856 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:56 dist-test-slave-4t1f krb5kdc[22430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:56 dist-test-slave-4t1f krb5kdc[22430](info): set up 2 sockets
Jun 19 06:32:56 dist-test-slave-4t1f krb5kdc[22430](info): commencing operation
krb5kdc: starting...
W20260619 06:32:57.105726 389 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:57 dist-test-slave-4t1f krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:57.342439 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36233
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:46317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:57.666929 22443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:57.667214 22443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:57.667382 22443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:57.674749 22443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:57.674854 22443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:57.674965 22443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:57.675081 22443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:57.675159 22443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:57.685053 22443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:46317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:57.686331 22443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:57.687641 22443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:57.699551 22449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:57.699761 22451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:57.700466 22448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:57.701894 22443 server_base.cc:1034] running on GCE node
I20260619 06:32:57.702826 22443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:57.704509 22443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:57.705745 22443 hybrid_clock.cc:648] HybridClock initialized: now 1781850777705689 us; error 46 us; skew 500 ppm
I20260619 06:32:57.706095 22443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:57 dist-test-slave-4t1f krb5kdc[22430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850777, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:57.708947 22443 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:57.710278 22443 webserver.cc:460] Webserver started at http://127.0.97.126:40057/ using document root <none> and password file <none>
I20260619 06:32:57.710903 22443 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:57.710994 22443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:57.711229 22443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:57.714370 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "3c9545ae2e5743e9bfc055c0868bc19e"
format_stamp: "Formatted at 2026-06-19 06:32:57 on dist-test-slave-4t1f"
server_key: "e0441131a60e8eb080d2102097589266"
server_key_iv: "774457d4efb89db42ca47b1fb901e06f"
server_key_version: "encryptionkey@0"
I20260619 06:32:57.715217 22443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "3c9545ae2e5743e9bfc055c0868bc19e"
format_stamp: "Formatted at 2026-06-19 06:32:57 on dist-test-slave-4t1f"
server_key: "e0441131a60e8eb080d2102097589266"
server_key_iv: "774457d4efb89db42ca47b1fb901e06f"
server_key_version: "encryptionkey@0"
I20260619 06:32:57.720068 22443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260619 06:32:57.723335 22458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:57.724692 22443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 06:32:57.724848 22443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "3c9545ae2e5743e9bfc055c0868bc19e"
format_stamp: "Formatted at 2026-06-19 06:32:57 on dist-test-slave-4t1f"
server_key: "e0441131a60e8eb080d2102097589266"
server_key_iv: "774457d4efb89db42ca47b1fb901e06f"
server_key_version: "encryptionkey@0"
I20260619 06:32:57.725014 22443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:57.749841 22443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:57.753988 22443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:57.754272 22443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:57.771975 22443 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36233
I20260619 06:32:57.772013 22509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36233 every 8 connection(s)
I20260619 06:32:57.773300 22443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:57.774850 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22443
I20260619 06:32:57.775041 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:57.775584 389 external_mini_cluster.cc:1428] Setting key ca6e3b1b8c24a49aaaf83a0abd72b84c
I20260619 06:32:57.777714 22510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Jun 19 06:32:57 dist-test-slave-4t1f krb5kdc[22430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:32:57.790804 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e: Bootstrap starting.
I20260619 06:32:57.794162 22510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:57.795131 22510 log.cc:824] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:57.797920 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:57.777416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32803 (local address 127.0.97.126:36233)
0619 06:32:57.778197 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:57.778209 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:57.778252 (+ 43us) server_negotiation.cc:407] Connection header received
0619 06:32:57.779864 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:57.779884 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:57.780271 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:57.780658 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:57.782422 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:57.783749 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:57.785590 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:57.786243 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:57.786374 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:57.791021 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:57.791030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:57.791044 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:57.791081 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:57.793993 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:57.794717 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:57.794723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:57.794728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:57.794878 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:57.795245 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:57.795252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:57.795255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:57.795798 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:57.796118 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:57.796539 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 06:32:57.796934 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":376,"threads_started":1}
I20260619 06:32:57.798015 22510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e: No bootstrap required, opened a new log
I20260619 06:32:57.801501 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:57.806021 22510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36233 } }
I20260619 06:32:57.806373 22510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:57.806476 22510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9545ae2e5743e9bfc055c0868bc19e, State: Initialized, Role: FOLLOWER
I20260619 06:32:57.807090 22510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36233 } }
I20260619 06:32:57.807330 22510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:57.807458 22510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:57.807616 22510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:57.809928 22510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36233 } }
I20260619 06:32:57.810410 22510 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9545ae2e5743e9bfc055c0868bc19e; no voters:
I20260619 06:32:57.811045 22510 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:57.811163 22520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:57.812320 22520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [term 1 LEADER]: Becoming Leader. State: Replica: 3c9545ae2e5743e9bfc055c0868bc19e, State: Running, Role: LEADER
I20260619 06:32:57.812912 22520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36233 } }
I20260619 06:32:57.813254 22510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:57.817369 22521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36233 } } }
I20260619 06:32:57.817453 22522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9545ae2e5743e9bfc055c0868bc19e. Latest consensus state: current_term: 1 leader_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9545ae2e5743e9bfc055c0868bc19e" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36233 } } }
I20260619 06:32:57.817766 22521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:57.817867 22522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:57.819489 22529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:57.822798 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:57.805536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58617 (local address 127.0.97.126:36233)
0619 06:32:57.805698 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:57.805706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:57.806103 (+ 397us) server_negotiation.cc:407] Connection header received
0619 06:32:57.806283 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:57.806288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:57.806397 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:57.806502 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:57.808086 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:57.808875 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:57.810151 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:57.810658 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:57.810793 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:57.813658 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:57.813663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:57.813666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:32:57.813693 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:57.817744 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:57.818734 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:57.818743 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:57.818749 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:57.819009 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:32:57.819532 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:32:57.819537 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:32:57.819541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:32:57.819959 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:57.820361 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:57.821988 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0619 06:32:57.822299 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 06:32:57.831435 22529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:57.840104 22529 catalog_manager.cc:1347] Generated new cluster ID: 021ab5790a864bb1b9bbc3d66a477c3b
I20260619 06:32:57.840205 22529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:57.849970 22529 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:57.851651 22529 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:57.858100 22529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9545ae2e5743e9bfc055c0868bc19e: Generated new TSK 0
I20260619 06:32:57.859018 22529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:57.867077 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22443
I20260619 06:32:57.868163 22505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:32:57.955122 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22443
2026-06-19T06:32:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1943 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:58 dist-test-slave-4t1f krb5kdc[22542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:58 dist-test-slave-4t1f krb5kdc[22542](info): set up 2 sockets
Jun 19 06:32:58 dist-test-slave-4t1f krb5kdc[22542](info): commencing operation
krb5kdc: starting...
W20260619 06:32:59.045742 389 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:32:59 dist-test-slave-4t1f krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:32:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:32:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:32:59.279798 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36421
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:36421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:32:59.602563 22555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:32:59.602836 22555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:32:59.603003 22555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:32:59.610067 22555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:32:59.610168 22555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:32:59.610297 22555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:32:59.610397 22555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:32:59.610459 22555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:32:59.620036 22555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:36421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:32:59.621310 22555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:32:59.622640 22555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:32:59.634079 22561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:59.634198 22560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:32:59.634124 22563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:32:59.635133 22555 server_base.cc:1034] running on GCE node
I20260619 06:32:59.635787 22555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:32:59.637053 22555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:32:59.638309 22555 hybrid_clock.cc:648] HybridClock initialized: now 1781850779638271 us; error 39 us; skew 500 ppm
I20260619 06:32:59.638648 22555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:32:59 dist-test-slave-4t1f krb5kdc[22542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850779, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:32:59.641655 22555 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:32:59.642967 22555 webserver.cc:460] Webserver started at http://127.0.97.126:41723/ using document root <none> and password file <none>
I20260619 06:32:59.643622 22555 fs_manager.cc:359] Metadata directory not provided
I20260619 06:32:59.643762 22555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:32:59.644089 22555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:32:59.646903 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "6d4c58b4cb954b4c9514a86846abd6e0"
format_stamp: "Formatted at 2026-06-19 06:32:59 on dist-test-slave-4t1f"
server_key: "a2ef8982a32b4b1c33ca59a5d0c04c6e"
server_key_iv: "1c2706fd5069e93fbdeafc7f0624ff46"
server_key_version: "encryptionkey@0"
I20260619 06:32:59.647631 22555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "6d4c58b4cb954b4c9514a86846abd6e0"
format_stamp: "Formatted at 2026-06-19 06:32:59 on dist-test-slave-4t1f"
server_key: "a2ef8982a32b4b1c33ca59a5d0c04c6e"
server_key_iv: "1c2706fd5069e93fbdeafc7f0624ff46"
server_key_version: "encryptionkey@0"
I20260619 06:32:59.652405 22555 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 06:32:59.655711 22570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:32:59.656888 22555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260619 06:32:59.657047 22555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "6d4c58b4cb954b4c9514a86846abd6e0"
format_stamp: "Formatted at 2026-06-19 06:32:59 on dist-test-slave-4t1f"
server_key: "a2ef8982a32b4b1c33ca59a5d0c04c6e"
server_key_iv: "1c2706fd5069e93fbdeafc7f0624ff46"
server_key_version: "encryptionkey@0"
I20260619 06:32:59.657262 22555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:32:59.673727 22555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:32:59.677461 22555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:32:59.677735 22555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:32:59.695632 22555 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:36421
I20260619 06:32:59.695664 22621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:36421 every 8 connection(s)
I20260619 06:32:59.696946 22555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:32:59.700551 22622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:32:59.702006 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22555
I20260619 06:32:59.702189 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:32:59.702610 389 external_mini_cluster.cc:1428] Setting key 88c5a3a88901613619e0738ffaea6644
I20260619 06:32:59.714926 22622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0: Bootstrap starting.
I20260619 06:32:59.716567 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:59.704251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56831 (local address 127.0.97.126:36421)
0619 06:32:59.705051 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:59.705084 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:59.705131 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:32:59.707006 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:59.707033 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:59.707545 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:59.707970 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:59.708449 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:59.710287 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:59.711523 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:59.712572 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:59.712761 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:59.713965 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:59.713976 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:59.713996 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:59.714693 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:59.714789 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:59.714817 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:59.714937 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:59.715098 (+ 161us) server_negotiation.cc:299] Negotiation successful
0619 06:32:59.715355 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":367,"threads_started":1}
I20260619 06:32:59.718470 22622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0: Neither blocks nor log segments found. Creating new log.
I20260619 06:32:59.719641 22622 log.cc:824] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0: Log is configured to *not* fsync() on all Append() calls
I20260619 06:32:59.720175 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:32:59.722600 22622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0: No bootstrap required, opened a new log
I20260619 06:32:59.731611 22625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:32:59.725127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38525 (local address 127.0.97.126:36421)
0619 06:32:59.725300 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 06:32:59.725311 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 06:32:59.725765 (+ 454us) server_negotiation.cc:407] Connection header received
0619 06:32:59.725908 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:32:59.725914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:32:59.725982 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:32:59.726093 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:32:59.726533 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:59.727615 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:59.729117 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:32:59.729799 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:32:59.729931 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:32:59.730887 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:32:59.730897 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:32:59.730902 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:32:59.730921 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:32:59.730965 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:32:59.730990 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:32:59.731083 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 06:32:59.731201 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 06:32:59.731281 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:32:59.732298 22622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36421 } }
I20260619 06:32:59.732663 22622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:32:59.732786 22622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4c58b4cb954b4c9514a86846abd6e0, State: Initialized, Role: FOLLOWER
I20260619 06:32:59.733394 22622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36421 } }
I20260619 06:32:59.733594 22622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:32:59.733692 22622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:32:59.733801 22622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:32:59.735780 22622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36421 } }
I20260619 06:32:59.736192 22622 leader_election.cc:302] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d4c58b4cb954b4c9514a86846abd6e0; no voters:
I20260619 06:32:59.736694 22622 leader_election.cc:288] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:32:59.736853 22632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:32:59.737658 22632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [term 1 LEADER]: Becoming Leader. State: Replica: 6d4c58b4cb954b4c9514a86846abd6e0, State: Running, Role: LEADER
I20260619 06:32:59.738191 22632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36421 } }
I20260619 06:32:59.738802 22622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:32:59.740887 22633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36421 } } }
I20260619 06:32:59.741287 22633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:59.741762 22634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d4c58b4cb954b4c9514a86846abd6e0. Latest consensus state: current_term: 1 leader_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d4c58b4cb954b4c9514a86846abd6e0" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 36421 } } }
I20260619 06:32:59.742060 22634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0 [sys.catalog]: This master's current role is: LEADER
I20260619 06:32:59.750943 22640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:32:59.756219 22640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:32:59.764726 22640 catalog_manager.cc:1347] Generated new cluster ID: 96b20c99b1cf4de59ff6715b7d2853f2
I20260619 06:32:59.764851 22640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:32:59.777885 22640 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:32:59.779148 22640 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:32:59.785493 22640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d4c58b4cb954b4c9514a86846abd6e0: Generated new TSK 0
I20260619 06:32:59.786252 22640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:32:59.799260 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22555
W20260619 06:32:59.800545 22617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56831
W20260619 06:32:59.801258 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22555: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:32:59.801401 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22555
2026-06-19T06:32:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1846 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:32:59 dist-test-slave-4t1f krb5kdc[22654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:32:59 dist-test-slave-4t1f krb5kdc[22654](info): set up 2 sockets
Jun 19 06:32:59 dist-test-slave-4t1f krb5kdc[22654](info): commencing operation
krb5kdc: starting...
W20260619 06:33:00.905493 389 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:33:01 dist-test-slave-4t1f krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:33:01.148587 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33567
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:43979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:33567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:01.484040 22667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:01.484404 22667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:01.484615 22667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:01.491941 22667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:01.492107 22667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:01.492228 22667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:01.492353 22667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:01.492440 22667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:01.502417 22667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:43979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:33567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:33567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:01.503763 22667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:01.505143 22667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:01.517501 22672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:01.517518 22673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:01.517539 22675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:01.518507 22667 server_base.cc:1034] running on GCE node
I20260619 06:33:01.519273 22667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:01.520646 22667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:01.521876 22667 hybrid_clock.cc:648] HybridClock initialized: now 1781850781521837 us; error 48 us; skew 500 ppm
I20260619 06:33:01.522246 22667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:01 dist-test-slave-4t1f krb5kdc[22654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850781, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:01.527917 22667 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:33:01.529421 22667 webserver.cc:460] Webserver started at http://127.0.97.126:36439/ using document root <none> and password file <none>
I20260619 06:33:01.530045 22667 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:01.530140 22667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:01.530385 22667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:01.533231 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "0181aaf7f404427bb8fe6c060abef760"
format_stamp: "Formatted at 2026-06-19 06:33:01 on dist-test-slave-4t1f"
server_key: "c8e371e72cba52ae7a72b268228e49c8"
server_key_iv: "3a03545a8660dbad47ffc464219102a5"
server_key_version: "encryptionkey@0"
I20260619 06:33:01.533933 22667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "0181aaf7f404427bb8fe6c060abef760"
format_stamp: "Formatted at 2026-06-19 06:33:01 on dist-test-slave-4t1f"
server_key: "c8e371e72cba52ae7a72b268228e49c8"
server_key_iv: "3a03545a8660dbad47ffc464219102a5"
server_key_version: "encryptionkey@0"
I20260619 06:33:01.538764 22667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 06:33:01.541980 22682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:01.543203 22667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:01.543381 22667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "0181aaf7f404427bb8fe6c060abef760"
format_stamp: "Formatted at 2026-06-19 06:33:01 on dist-test-slave-4t1f"
server_key: "c8e371e72cba52ae7a72b268228e49c8"
server_key_iv: "3a03545a8660dbad47ffc464219102a5"
server_key_version: "encryptionkey@0"
I20260619 06:33:01.543562 22667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:01.549201 22667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:01.552879 22667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:01.553241 22667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:01.571250 22667 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:33567
I20260619 06:33:01.571328 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:33567 every 8 connection(s)
I20260619 06:33:01.572547 22667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:01.576045 22734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:01.582324 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22667
I20260619 06:33:01.582499 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:33:01.583015 389 external_mini_cluster.cc:1428] Setting key e2c95bcd069078845058984208a463e2
I20260619 06:33:01.586522 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760: Bootstrap starting.
I20260619 06:33:01.589987 22734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:01.590760 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:01.584799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50771 (local address 127.0.97.126:33567)
0619 06:33:01.585640 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:01.585652 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:01.585692 (+ 40us) server_negotiation.cc:407] Connection header received
0619 06:33:01.587331 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:01.587352 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:01.587666 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:01.587972 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:01.589004 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:01.589010 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:01.589022 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:33:01.589035 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:01.589145 (+ 110us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:33:01.589166 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:01.589309 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:01.589512 (+ 203us) server_negotiation.cc:299] Negotiation successful
0619 06:33:01.589839 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":416,"threads_started":1}
I20260619 06:33:01.591063 22734 log.cc:824] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:01.594307 22734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760: No bootstrap required, opened a new log
I20260619 06:33:01.594520 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:33:01.601814 22734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0181aaf7f404427bb8fe6c060abef760" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33567 } }
I20260619 06:33:01.602268 22734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 06:33:01.602370 22734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0181aaf7f404427bb8fe6c060abef760, State: Initialized, Role: FOLLOWER
I20260619 06:33:01.602977 22734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0181aaf7f404427bb8fe6c060abef760" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33567 } }
I20260619 06:33:01.603222 22734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 06:33:01.603358 22734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 06:33:01.603565 22734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:01.603647 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:01.598584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35963 (local address 127.0.97.126:33567)
0619 06:33:01.598761 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:01.598767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:01.601623 (+ 2856us) server_negotiation.cc:407] Connection header received
0619 06:33:01.601788 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:01.601791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:01.601836 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:01.601912 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:01.602920 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:01.602925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:01.602930 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 06:33:01.602946 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:01.602994 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 06:33:01.603008 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:01.603140 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:01.603233 (+ 93us) server_negotiation.cc:299] Negotiation successful
0619 06:33:01.603287 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 06:33:01.605906 22734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0181aaf7f404427bb8fe6c060abef760" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33567 } }
I20260619 06:33:01.606370 22734 leader_election.cc:302] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0181aaf7f404427bb8fe6c060abef760; no voters:
I20260619 06:33:01.606920 22734 leader_election.cc:288] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 06:33:01.607004 22744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:01.607676 22744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [term 1 LEADER]: Becoming Leader. State: Replica: 0181aaf7f404427bb8fe6c060abef760, State: Running, Role: LEADER
I20260619 06:33:01.608204 22744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0181aaf7f404427bb8fe6c060abef760" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33567 } }
I20260619 06:33:01.608898 22734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:01.610816 22746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0181aaf7f404427bb8fe6c060abef760. Latest consensus state: current_term: 1 leader_uuid: "0181aaf7f404427bb8fe6c060abef760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0181aaf7f404427bb8fe6c060abef760" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33567 } } }
I20260619 06:33:01.611025 22745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0181aaf7f404427bb8fe6c060abef760" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0181aaf7f404427bb8fe6c060abef760" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 33567 } } }
I20260619 06:33:01.611254 22746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:01.611279 22745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:01.613597 22749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:01.619973 22749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:01.628547 22749 catalog_manager.cc:1347] Generated new cluster ID: 6ac31a2f0eca412f96c8a1e6429ac2ef
I20260619 06:33:01.628674 22749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:01.646561 22749 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:01.647027 22749 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:01.662905 22749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0181aaf7f404427bb8fe6c060abef760: Generated new TSK 0
I20260619 06:33:01.663851 22749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:33:01.671330 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22667
W20260619 06:33:01.672667 22729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50771
W20260619 06:33:01.673503 389 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260619 06:33:01.673645 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22667
2026-06-19T06:33:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1874 ms)
[----------] 18 tests from AuthTokenIssuingTest (34149 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:01 dist-test-slave-4t1f krb5kdc[22765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:01 dist-test-slave-4t1f krb5kdc[22765](info): set up 2 sockets
Jun 19 06:33:01 dist-test-slave-4t1f krb5kdc[22765](info): commencing operation
krb5kdc: starting...
W20260619 06:33:02.781538 389 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:33:02 dist-test-slave-4t1f krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:33:03.032446 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45487
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45487,127.0.97.125:36795,127.0.97.124:44703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:03.363320 22778 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:33:03.363694 22778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:03.363831 22778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:03.363981 22778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:03.370886 22778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:03.371016 22778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:03.371155 22778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:03.371243 22778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:03.371361 22778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:03.381052 22778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45487,127.0.97.125:36795,127.0.97.124:44703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:45487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:03.382498 22778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:03.383838 22778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:03.395841 22783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:03.395815 22786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:03.396327 22784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:03.396991 22778 server_base.cc:1034] running on GCE node
I20260619 06:33:03.397775 22778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:03.399147 22778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:03.400429 22778 hybrid_clock.cc:648] HybridClock initialized: now 1781850783400382 us; error 37 us; skew 500 ppm
I20260619 06:33:03.400771 22778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:03 dist-test-slave-4t1f krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:03.403923 22778 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:33:03.405504 22778 webserver.cc:460] Webserver started at http://127.0.97.126:39131/ using document root <none> and password file <none>
I20260619 06:33:03.406174 22778 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:03.406311 22778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:03.406638 22778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:03.409548 22778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "416cc7b19cb74877b3c2f7a99f69464a"
format_stamp: "Formatted at 2026-06-19 06:33:03 on dist-test-slave-4t1f"
server_key: "a67388763a59c572c1fdcac513ca97af"
server_key_iv: "34dc85de62f8b1b8fe4cdbde1cb0291e"
server_key_version: "encryptionkey@0"
I20260619 06:33:03.410348 22778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "416cc7b19cb74877b3c2f7a99f69464a"
format_stamp: "Formatted at 2026-06-19 06:33:03 on dist-test-slave-4t1f"
server_key: "a67388763a59c572c1fdcac513ca97af"
server_key_iv: "34dc85de62f8b1b8fe4cdbde1cb0291e"
server_key_version: "encryptionkey@0"
I20260619 06:33:03.415556 22778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 06:33:03.419021 22793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:03.420295 22778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 06:33:03.420497 22778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "416cc7b19cb74877b3c2f7a99f69464a"
format_stamp: "Formatted at 2026-06-19 06:33:03 on dist-test-slave-4t1f"
server_key: "a67388763a59c572c1fdcac513ca97af"
server_key_iv: "34dc85de62f8b1b8fe4cdbde1cb0291e"
server_key_version: "encryptionkey@0"
I20260619 06:33:03.420734 22778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:03.439617 22778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:03.443439 22778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:03.443719 22778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:03.462350 22844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:45487 every 8 connection(s)
I20260619 06:33:03.462347 22778 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:45487
I20260619 06:33:03.463691 22778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:03.465638 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22778
I20260619 06:33:03.465857 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:33:03.466631 389 external_mini_cluster.cc:1428] Setting key 8c59a25c1073ef58ebd7e0ef39e0bd85
I20260619 06:33:03.467780 22845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:03.477396 22845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Jun 19 06:33:03 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
I20260619 06:33:03.500319 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.480353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45487 (local address 127.0.0.1:47392)
0619 06:33:03.481354 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.481398 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:03.483292 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:03.484065 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:03.484071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:03.484758 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:03.485119 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:03.485149 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.486928 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.486933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:03.488134 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:03.488147 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.489105 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.489113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:03.489408 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:03.490071 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:03.490092 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:03.492195 (+ 2103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:03.496397 (+ 4202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:03.496407 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:03.496416 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:03.496781 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:03.497325 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:03.497329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:03.497331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:03.497479 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:03.498488 (+ 1009us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:03.498493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:03.498692 (+ 199us) client_negotiation.cc:769] Sending connection context
0619 06:33:03.498893 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 06:33:03.499282 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":53,"thread_start_us":513,"threads_started":1}
I20260619 06:33:03.500356 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.480370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.97.126:45487)
0619 06:33:03.482126 (+ 1756us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:03.482136 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:03.482189 (+ 53us) server_negotiation.cc:407] Connection header received
0619 06:33:03.483633 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:03.483638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:03.483851 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:03.484058 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:03.485331 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.486753 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.488289 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.488837 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.489035 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:03.492448 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:03.492459 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:03.492473 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:03.492513 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:03.496243 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:03.496932 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:03.496943 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:03.496949 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:03.497208 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:03.497601 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:03.497609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:03.497614 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:03.498360 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:03.498550 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:03.499102 (+ 552us) server_negotiation.cc:299] Negotiation successful
0619 06:33:03.499458 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":1559,"thread_start_us":1372,"threads_started":1}
I20260619 06:33:03.503263 22845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:33:03.504303 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.503833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33088)
0619 06:33:03.504015 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.504149 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:33:03.504572 22797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
I20260619 06:33:03.506209 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.505783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33090)
0619 06:33:03.505956 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.506046 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:33:03.506433 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36795: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260619 06:33:03.524786 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:36795
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45487,127.0.97.125:36795,127.0.97.124:44703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:33:03.555783 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } attempt: 1
I20260619 06:33:03.556957 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.556476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33102)
0619 06:33:03.556685 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.556785 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:33:03.557937 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.557580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33112)
0619 06:33:03.557725 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.557804 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:33:03.558142 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36795: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
I20260619 06:33:03.626401 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } attempt: 2
I20260619 06:33:03.627681 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.627028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33122)
0619 06:33:03.627328 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.627457 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 06:33:03.628834 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.628338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33134)
0619 06:33:03.628526 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.628643 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:33:03.629051 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36795: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
I20260619 06:33:03.720377 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } attempt: 3
I20260619 06:33:03.721671 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.721216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33148)
0619 06:33:03.721424 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.721529 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:33:03.722828 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.722504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33160)
0619 06:33:03.722641 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.722707 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:33:03.723101 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36795: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
W20260619 06:33:03.840126 22852 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:33:03.840404 22852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:03.840489 22852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:03.840649 22852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:03.847651 22852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:03.847733 22852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:03.847847 22852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:03.847921 22852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:03.848024 22852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:03.857424 22852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45487,127.0.97.125:36795,127.0.97.124:44703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:03.858636 22852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:03.859879 22852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:33:03.866328 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } attempt: 4
I20260619 06:33:03.867444 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.866974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33176)
0619 06:33:03.867243 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.867326 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 06:33:03.868394 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.868052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33184)
0619 06:33:03.868208 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.868278 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:33:03.868592 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:36795: Network error: Client connection negotiation failed: client connection to 127.0.97.125:36795: connect: Connection refused (error 111)
W20260619 06:33:03.871316 22858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:03.871469 22857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:03.871454 22860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:03.872412 22852 server_base.cc:1034] running on GCE node
I20260619 06:33:03.873152 22852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:03.874408 22852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:03.875641 22852 hybrid_clock.cc:648] HybridClock initialized: now 1781850783875589 us; error 48 us; skew 500 ppm
I20260619 06:33:03.875949 22852 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:03 dist-test-slave-4t1f krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:03.879053 22852 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260619 06:33:03.880477 22852 webserver.cc:460] Webserver started at http://127.0.97.125:36207/ using document root <none> and password file <none>
I20260619 06:33:03.881356 22852 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:03.881475 22852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:03.881767 22852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:03.884553 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data/instance:
uuid: "f94ef3da1168450783ca62324397947c"
format_stamp: "Formatted at 2026-06-19 06:33:03 on dist-test-slave-4t1f"
server_key: "4d4d32a4bc9221ad5ecc814e2236a4bd"
server_key_iv: "cca77b99e9c01d5e9d949a5fb5a59745"
server_key_version: "encryptionkey@0"
I20260619 06:33:03.885360 22852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/wal/instance:
uuid: "f94ef3da1168450783ca62324397947c"
format_stamp: "Formatted at 2026-06-19 06:33:03 on dist-test-slave-4t1f"
server_key: "4d4d32a4bc9221ad5ecc814e2236a4bd"
server_key_iv: "cca77b99e9c01d5e9d949a5fb5a59745"
server_key_version: "encryptionkey@0"
I20260619 06:33:03.890184 22852 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 06:33:03.893518 22867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:03.894847 22852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260619 06:33:03.895046 22852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/wal
uuid: "f94ef3da1168450783ca62324397947c"
format_stamp: "Formatted at 2026-06-19 06:33:03 on dist-test-slave-4t1f"
server_key: "4d4d32a4bc9221ad5ecc814e2236a4bd"
server_key_iv: "cca77b99e9c01d5e9d949a5fb5a59745"
server_key_version: "encryptionkey@0"
I20260619 06:33:03.895293 22852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:03.908550 22852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:03.912911 22852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:03.913241 22852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:03.932296 22852 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:36795
I20260619 06:33:03.932314 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:36795 every 8 connection(s)
I20260619 06:33:03.933781 22852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
I20260619 06:33:03.937008 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22852
I20260619 06:33:03.937247 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-1/wal/instance
I20260619 06:33:03.937309 22919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:03.937705 389 external_mini_cluster.cc:1428] Setting key 6767188e96b80b8774e6ab64081c8e97
I20260619 06:33:03.943984 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:03 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
I20260619 06:33:03.964613 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.946584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47394 (local address 127.0.97.126:45487)
0619 06:33:03.946781 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:03.946788 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:03.947923 (+ 1135us) server_negotiation.cc:407] Connection header received
0619 06:33:03.949769 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:03.949778 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:03.949866 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:03.950010 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:03.951364 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.952520 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.954016 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.954757 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.954905 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:03.958096 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:03.958105 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:03.958112 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:03.958161 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:03.961647 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:03.962340 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:03.962345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:03.962347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:03.962459 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:03.962853 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:03.962857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:03.962860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:03.963190 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:03.963347 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:03.964090 (+ 743us) server_negotiation.cc:299] Negotiation successful
0619 06:33:03.964312 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 06:33:03.965605 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.946999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45487 (local address 127.0.0.1:47394)
0619 06:33:03.947753 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.947801 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:03.949518 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:03.950191 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:03.950198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:03.950662 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:03.951173 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:03.951189 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.952677 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.952680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:03.953868 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:03.953880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.954949 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.954954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:03.955143 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:03.955797 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:03.955817 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:03.957862 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:03.961763 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:03.961769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:03.961779 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:03.962204 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:03.962567 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:03.962573 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:03.962577 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:03.962734 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:03.963323 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:03.963330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:03.963696 (+ 366us) client_negotiation.cc:769] Sending connection context
0619 06:33:03.963977 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 06:33:03.964455 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":349,"threads_started":1}
I20260619 06:33:03.967350 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:03 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:33:03.983552 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.967970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33200)
0619 06:33:03.968256 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.968289 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:03.968411 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:03.969557 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:03.969561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:03.969927 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:03.970074 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:03.970083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.972143 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.972149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:03.973812 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:03.973831 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.974615 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.974624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:03.974890 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:03.975503 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:03.975533 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:03.978092 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:03.981202 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:03.981207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:03.981211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:03.981555 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:03.981919 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:03.981925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:03.981930 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:03.982061 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:03.982595 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:03.982604 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:03.982724 (+ 120us) client_negotiation.cc:769] Sending connection context
0619 06:33:03.982906 (+ 182us) client_negotiation.cc:240] Negotiation successful
0619 06:33:03.983174 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260619 06:33:03.983671 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.968291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.0.97.125:36795)
0619 06:33:03.969008 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:03.969018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:03.969049 (+ 31us) server_negotiation.cc:407] Connection header received
0619 06:33:03.969124 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:03.969135 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:03.969384 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:03.969543 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:03.970250 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.971980 (+ 1730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.973960 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:03.974458 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:03.974550 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:03.978312 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:03.978322 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:03.978334 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:03.978361 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:03.981008 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:03.981688 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:03.981696 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:03.981699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:03.981819 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:03.982164 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:03.982168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:03.982171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:03.982471 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:03.982665 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:03.983031 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 06:33:03.983286 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":533,"thread_start_us":382,"threads_started":1}
I20260619 06:33:03.985785 22919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:33:03.986984 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.986543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38276)
0619 06:33:03.986721 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.986845 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 06:33:03.987282 22870 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
I20260619 06:33:03.988792 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:03.988457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38288)
0619 06:33:03.988579 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:03.988661 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:33:03.989113 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260619 06:33:03.994736 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:44703
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:40485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:45487,127.0.97.125:36795,127.0.97.124:44703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:33:04.038424 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 1
I20260619 06:33:04.039508 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.039077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38302)
0619 06:33:04.039267 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.039371 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:33:04.040512 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.040163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38308)
0619 06:33:04.040338 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.040418 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 06:33:04.040804 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
I20260619 06:33:04.109041 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 2
I20260619 06:33:04.110198 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.109785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38314)
0619 06:33:04.109976 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.110060 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 06:33:04.111120 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.110809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38322)
0619 06:33:04.110942 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.111015 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260619 06:33:04.111308 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
I20260619 06:33:04.167861 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } attempt: 5
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:33:04.181016 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.168648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33214)
0619 06:33:04.168845 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.168870 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:04.169026 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:04.169422 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:04.169429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:04.169833 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:04.170026 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.170035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.171067 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.171073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.171927 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.171938 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.172698 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.172704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.172911 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.173602 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:04.173620 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:04.175603 (+ 1983us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:04.178638 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.178643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.178647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.179015 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.179380 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.179386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.179390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.179555 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.180048 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:04.180053 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:04.180209 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 06:33:04.180375 (+ 166us) client_negotiation.cc:240] Negotiation successful
0619 06:33:04.180646 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 06:33:04.181006 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.168779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33214 (local address 127.0.97.125:36795)
0619 06:33:04.169001 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.169006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.169028 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:33:04.169208 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.169212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.169274 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.169382 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.170180 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.170882 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.172084 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.172540 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.172678 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.175792 (+ 3114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.175797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.175801 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.175830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.178489 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.179132 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.179140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.179146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.179267 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.179660 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.179666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.179670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.179930 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.180080 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.180490 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.180696 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:33:04.182348 22845 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:33:04.183460 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.183101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38324)
0619 06:33:04.183254 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.183326 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260619 06:33:04.184471 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.184151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38338)
0619 06:33:04.184302 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.184380 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 06:33:04.184757 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
I20260619 06:33:04.202535 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 3
I20260619 06:33:04.203493 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.203072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38346)
0619 06:33:04.203262 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.203353 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 06:33:04.204423 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.204064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38350)
0619 06:33:04.204266 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.204327 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260619 06:33:04.204596 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
I20260619 06:33:04.235968 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 1
I20260619 06:33:04.237068 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.236619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38352)
0619 06:33:04.236811 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.236903 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:33:04.238101 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.237831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38354)
0619 06:33:04.237951 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.238012 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:33:04.238401 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
I20260619 06:33:04.306612 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 2
I20260619 06:33:04.307785 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.307286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38362)
0619 06:33:04.307470 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.307561 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260619 06:33:04.308919 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.308619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38368)
0619 06:33:04.308753 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.308817 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:33:04.309124 22845 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
W20260619 06:33:04.315490 22926 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:33:04.315814 22926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:04.315932 22926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:04.316063 22926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:04.323191 22926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:04.323308 22926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:04.323470 22926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:04.323561 22926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:04.323657 22926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:04.333775 22926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:40485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:45487,127.0.97.125:36795,127.0.97.124:44703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:44703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:04.335062 22926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:04.336329 22926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:04.347306 22931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:04.347281 22932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:04.347236 22934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:04.347767 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 4
I20260619 06:33:04.348973 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.348418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38370)
0619 06:33:04.348691 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.348787 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 06:33:04.349752 22926 server_base.cc:1034] running on GCE node
I20260619 06:33:04.350252 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.349696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38384)
0619 06:33:04.350022 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.350126 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216}
W20260619 06:33:04.350518 22919 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:44703: Network error: Client connection negotiation failed: client connection to 127.0.97.124:44703: connect: Connection refused (error 111)
I20260619 06:33:04.350584 22926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:04.351943 22926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:04.353204 22926 hybrid_clock.cc:648] HybridClock initialized: now 1781850784353135 us; error 40 us; skew 500 ppm
I20260619 06:33:04.353547 22926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:04.356529 22926 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260619 06:33:04.357918 22926 webserver.cc:460] Webserver started at http://127.0.97.124:40971/ using document root <none> and password file <none>
I20260619 06:33:04.358585 22926 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:04.358711 22926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:04.359015 22926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:04.361867 22926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data/instance:
uuid: "5094ee875dc0494d826a491025a7335e"
format_stamp: "Formatted at 2026-06-19 06:33:04 on dist-test-slave-4t1f"
server_key: "b1eb7715cf61b9e440910e288df38ec6"
server_key_iv: "b6fc4b841d36711e98b01fcab9218d8b"
server_key_version: "encryptionkey@0"
I20260619 06:33:04.362612 22926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/wal/instance:
uuid: "5094ee875dc0494d826a491025a7335e"
format_stamp: "Formatted at 2026-06-19 06:33:04 on dist-test-slave-4t1f"
server_key: "b1eb7715cf61b9e440910e288df38ec6"
server_key_iv: "b6fc4b841d36711e98b01fcab9218d8b"
server_key_version: "encryptionkey@0"
I20260619 06:33:04.367406 22926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 06:33:04.370540 22941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:04.371749 22926 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260619 06:33:04.371923 22926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/wal
uuid: "5094ee875dc0494d826a491025a7335e"
format_stamp: "Formatted at 2026-06-19 06:33:04 on dist-test-slave-4t1f"
server_key: "b1eb7715cf61b9e440910e288df38ec6"
server_key_iv: "b6fc4b841d36711e98b01fcab9218d8b"
server_key_version: "encryptionkey@0"
I20260619 06:33:04.372124 22926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:04.390350 22926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:04.394166 22926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:04.394416 22926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:04.412310 22926 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:44703
I20260619 06:33:04.412331 22992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:44703 every 8 connection(s)
I20260619 06:33:04.413618 22926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
I20260619 06:33:04.415328 22845 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 3
I20260619 06:33:04.416234 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 22926
I20260619 06:33:04.416424 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781850433974487-389-0/minicluster-data/master-2/wal/instance
I20260619 06:33:04.416927 389 external_mini_cluster.cc:1428] Setting key 9bc15d3fe54b93ce6abb2402a7d9a4ec
I20260619 06:33:04.417640 22993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:04.426702 22993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:33:04.438444 22848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.415994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38396)
0619 06:33:04.416300 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.416320 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:04.416435 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:04.421304 (+ 4869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:04.421311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:04.421812 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:04.421995 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.422007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.424551 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.424560 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.426031 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.426047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.427172 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.427179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.427372 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.428065 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:04.428082 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:04.430501 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:04.435222 (+ 4721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.435228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.435232 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.435808 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.436245 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.436252 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.436255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.436421 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.437321 (+ 900us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:04.437327 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:04.437510 (+ 183us) client_negotiation.cc:769] Sending connection context
0619 06:33:04.437704 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 06:33:04.438003 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154}
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:33:04.439589 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.418277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.0.97.124:44703)
0619 06:33:04.421049 (+ 2772us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.421054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.421098 (+ 44us) server_negotiation.cc:407] Connection header received
0619 06:33:04.421167 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.421172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.421240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.422955 (+ 1715us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.423246 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.424580 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.426106 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.426884 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.427071 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.431781 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.431791 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.431796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.431837 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.435533 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.436325 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.436330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.436334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.436489 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.436938 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.436943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.436947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.437358 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.437661 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.438042 (+ 381us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.438578 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":2627,"spinlock_wait_cycles":15232,"thread_start_us":2453,"threads_started":1}
I20260619 06:33:04.439615 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.416953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.0.97.124:44703)
0619 06:33:04.417959 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.417973 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.418019 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:33:04.420327 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.420351 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.420855 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.421308 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.422187 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.424402 (+ 2215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.426220 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.426977 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.427647 (+ 670us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.430795 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.430802 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.430813 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.430843 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.435097 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.435959 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.435967 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.435981 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.436128 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.436553 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.436558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.436562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.437172 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.437355 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.438877 (+ 1522us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.439097 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":779,"thread_start_us":437,"threads_started":1}
I20260619 06:33:04.446097 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.429556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:45487 (local address 127.0.0.1:47400)
0619 06:33:04.430292 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.430345 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:04.430551 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:04.431085 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:04.431097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:04.431924 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:04.432236 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.432247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.433526 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.433531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.434846 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.434858 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.435856 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.435862 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.436056 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.436649 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:04.436670 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:04.439480 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:04.443387 (+ 3907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.443396 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.443403 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.443910 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.444344 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.444351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.444356 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.444484 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.445014 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:04.445021 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:04.445203 (+ 182us) client_negotiation.cc:769] Sending connection context
0619 06:33:04.445374 (+ 171us) client_negotiation.cc:240] Negotiation successful
0619 06:33:04.445679 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":402,"threads_started":1}
I20260619 06:33:04.447079 22849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.429593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47400 (local address 127.0.97.126:45487)
0619 06:33:04.429793 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.429800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.430436 (+ 636us) server_negotiation.cc:407] Connection header received
0619 06:33:04.430717 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.430722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.430790 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.430856 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.432384 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.433355 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.434990 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.435670 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.435798 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.439688 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.439695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.439700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.439734 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.443239 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.444048 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.444055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.444064 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.444218 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.444594 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.444600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.444604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.444889 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.445043 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.446600 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.446819 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 06:33:04.447436 22845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a: Bootstrap starting.
I20260619 06:33:04.448145 22993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:33:04.451968 22845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a: Neither blocks nor log segments found. Creating new log.
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:33:04.453230 22845 log.cc:824] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:04.456213 22845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a: No bootstrap required, opened a new log
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:33:04.460897 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.442726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.0.97.126:45487)
0619 06:33:04.443297 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.443301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.443577 (+ 276us) server_negotiation.cc:407] Connection header received
0619 06:33:04.443701 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.443711 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.443788 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.443897 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.445410 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.446644 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.448655 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.449814 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.449952 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.453990 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.454001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.454006 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.454051 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.457645 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.458693 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.458699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.458702 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.458859 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.459263 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.459271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.459275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.459698 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.459983 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.460278 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.460491 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":257,"threads_started":1}
I20260619 06:33:04.463970 22923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.449208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33226 (local address 127.0.97.125:36795)
0619 06:33:04.449365 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.449369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.449384 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:33:04.449428 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.449431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.449484 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.449559 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.450458 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.451467 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.453256 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.453920 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.454034 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.457634 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.457640 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.457645 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.457684 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.460845 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.461677 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.461682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.461684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.461858 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.462326 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.462330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.462334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.462672 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.462809 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.463330 (+ 521us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.463564 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 06:33:04.464748 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.448740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:36795 (local address 127.0.0.1:33226)
0619 06:33:04.448908 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.448927 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:04.449025 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:04.449627 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:04.449634 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:04.450092 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:04.450275 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.450285 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.451665 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.451669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.453051 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.453097 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.454217 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.454221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.454355 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.454861 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:04.454888 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:04.457427 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:04.460979 (+ 3552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.460984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.460988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.461535 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.462015 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.462019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.462023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.462198 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.462891 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:04.462896 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:04.463047 (+ 151us) client_negotiation.cc:769] Sending connection context
0619 06:33:04.464136 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0619 06:33:04.464398 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260619 06:33:04.465809 22993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:33:04.477834 22845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
I20260619 06:33:04.478253 22845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416cc7b19cb74877b3c2f7a99f69464a, State: Initialized, Role: FOLLOWER
I20260619 06:33:04.478763 22845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
I20260619 06:33:04.479872 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.463146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.0.97.125:36795)
0619 06:33:04.463762 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.463768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.463790 (+ 22us) server_negotiation.cc:407] Connection header received
0619 06:33:04.463915 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.463920 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.463993 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.464109 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.465875 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.467056 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.468425 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.469025 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.469194 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.473719 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.473725 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.473728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.473755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.476821 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.477572 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.477577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.477582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.477737 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.478145 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.478153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.478157 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.478508 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.478767 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.479049 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.479278 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":311,"threads_started":1}
I20260619 06:33:04.481098 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.466598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38422)
0619 06:33:04.466827 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.466842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:04.466941 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:04.467243 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:04.467248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:04.467672 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:04.467844 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.467854 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.469187 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.469195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.470657 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.470670 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.471660 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.471665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.471896 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.472423 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:04.472448 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:04.474847 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:04.478439 (+ 3592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.478443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.478447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.478920 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.479307 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.479311 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.479314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.479447 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.480112 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:04.480116 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:04.480275 (+ 159us) client_negotiation.cc:769] Sending connection context
0619 06:33:04.480426 (+ 151us) client_negotiation.cc:240] Negotiation successful
0619 06:33:04.480721 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260619 06:33:04.481570 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.466557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.0.97.124:44703)
0619 06:33:04.466729 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.466734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.466917 (+ 183us) server_negotiation.cc:407] Connection header received
0619 06:33:04.467037 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.467041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.467110 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.467203 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.467985 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.469008 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.470807 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.471511 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.471668 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.475024 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.475033 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.475040 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.475075 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.478308 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.479036 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.479040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.479043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.479190 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.479552 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.479556 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.479559 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.479910 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.480062 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.480498 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.480971 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":149,"server-negotiator.queue_time_us":54}
I20260619 06:33:04.483381 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.483776 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:04.485144 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:33:04.485419 22845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:04.486521 22993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e: Bootstrap starting.
I20260619 06:33:04.488476 22820 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "416cc7b19cb74877b3c2f7a99f69464a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47412
I20260619 06:33:04.488897 22820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:33:04.489115 22820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:04.490420 22993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:04.491679 22993 log.cc:824] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:04.492750 22820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
I20260619 06:33:04.495054 22993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e: No bootstrap required, opened a new log
I20260619 06:33:04.495260 22820 leader_election.cc:288] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [CANDIDATE]: Term 1 election: Requested vote from peers f94ef3da1168450783ca62324397947c (127.0.97.125:36795), 5094ee875dc0494d826a491025a7335e (127.0.97.124:44703)
I20260619 06:33:04.507812 22993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
I20260619 06:33:04.508291 22993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5094ee875dc0494d826a491025a7335e, State: Initialized, Role: FOLLOWER
I20260619 06:33:04.508781 22993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
W20260619 06:33:04.513293 23014 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:33:04.513741 23014 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:33:04.514310 23017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.515560 22993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:04.515651 22968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "416cc7b19cb74877b3c2f7a99f69464a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5094ee875dc0494d826a491025a7335e"
I20260619 06:33:04.522557 22968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:04.498924 22778 master_runner.cc:186] Error getting master registration for 127.0.97.125:36795: OK, instance_id { permanent_uuid: "f94ef3da1168450783ca62324397947c" instance_seqno: 1781850783927058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:33:04.523164 22778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:33:04.525985 22968 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416cc7b19cb74877b3c2f7a99f69464a in term 1.
I20260619 06:33:04.526861 22796 leader_election.cc:302] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416cc7b19cb74877b3c2f7a99f69464a, 5094ee875dc0494d826a491025a7335e; no voters:
I20260619 06:33:04.527383 23002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:04.528038 23002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [term 1 LEADER]: Becoming Leader. State: Replica: 416cc7b19cb74877b3c2f7a99f69464a, State: Running, Role: LEADER
I20260619 06:33:04.528620 23002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
I20260619 06:33:04.534179 23017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:04.542551 23021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416cc7b19cb74877b3c2f7a99f69464a. Latest consensus state: current_term: 1 leader_uuid: "416cc7b19cb74877b3c2f7a99f69464a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.542989 23021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:04.547003 23025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:04.549247 23025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:04.566268 22968 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [term 1 FOLLOWER]: Refusing update from remote peer 416cc7b19cb74877b3c2f7a99f69464a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:04.567272 23021 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:04.596827 23017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416cc7b19cb74877b3c2f7a99f69464a. Latest consensus state: current_term: 1 leader_uuid: "416cc7b19cb74877b3c2f7a99f69464a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.597235 23017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:04.601313 23017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "416cc7b19cb74877b3c2f7a99f69464a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.602208 23017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:04.602348 23027 mvcc.cc:205] Tried to move back new op lower bound from 7298460813566914560 to 7298460813434560512. Current Snapshot: MvccSnapshot[applied={T|T < 7298460813566914560}]
I20260619 06:33:04.609051 23025 catalog_manager.cc:1347] Generated new cluster ID: 4a8e1199f94b4818b0f91f1515a283f8
I20260619 06:33:04.609216 23025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:04.609560 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "416cc7b19cb74877b3c2f7a99f69464a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.609855 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:04.610241 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.571562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.0.97.125:36795)
0619 06:33:04.571750 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.571756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.585731 (+ 13975us) server_negotiation.cc:407] Connection header received
0619 06:33:04.585996 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.586005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.586112 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.586233 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.588510 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.589951 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.591913 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.592640 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.592760 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.596027 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.596036 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.596041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.596087 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.600052 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.601157 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.601165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.601171 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.601396 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.601940 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.601948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.601953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.602360 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.602628 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.609684 (+ 7056us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.609920 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:33:04.617807 23032 mvcc.cc:205] Tried to move back new op lower bound from 7298460813566914560 to 7298460813434560512. Current Snapshot: MvccSnapshot[applied={T|T < 7298460813566914560 or (T in {7298460813566914560})}]
I20260619 06:33:04.622723 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.559754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.0.97.126:45487)
0619 06:33:04.559988 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.559994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.564204 (+ 4210us) server_negotiation.cc:407] Connection header received
0619 06:33:04.564455 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.564464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.564583 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.564723 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.578119 (+ 13396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.579330 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.584618 (+ 5288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.585407 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.585526 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.611335 (+ 25809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.611342 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.611348 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.611392 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.615115 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.618141 (+ 3026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.618160 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.618164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.618316 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.620849 (+ 2533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.620854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.620858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.621257 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.621509 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.622084 (+ 575us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.622320 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:33:04.631772 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.559941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.0.97.124:44703)
0619 06:33:04.560146 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.560153 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.572696 (+ 12543us) server_negotiation.cc:407] Connection header received
0619 06:33:04.573217 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.573226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.573357 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.573445 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.593693 (+ 20248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.594917 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.615929 (+ 21012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.616645 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.616770 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.624899 (+ 8129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.624906 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.624911 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.624949 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.628690 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.629515 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.629521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.629525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.629681 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.630091 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.630097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.630100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.630468 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.630686 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.631192 (+ 506us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.631431 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:33:04.650091 22919 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } attempt: 5
I20260619 06:33:04.654042 23025 catalog_manager.cc:1370] Generated new certificate authority record
Jun 19 06:33:04 dist-test-slave-4t1f krb5kdc[22765](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:33:04.664604 23042 catalog_manager.cc:1259] Loaded cluster ID: 4a8e1199f94b4818b0f91f1515a283f8
I20260619 06:33:04.664752 23042 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e: loading cluster ID for follower catalog manager: success
I20260619 06:33:04.669860 23025 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:04.673249 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.650748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:44703 (local address 127.0.0.1:38436)
0619 06:33:04.650902 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:04.650920 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:04.651017 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:04.653151 (+ 2134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:04.653155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:04.653731 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:04.653934 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.653948 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.657323 (+ 3375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.657329 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.658753 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:04.658768 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.660109 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.660115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:04.660332 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.661100 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:04.661128 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:04.664002 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:04.669804 (+ 5802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.669810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.669816 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.670351 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.671029 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:04.671035 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:04.671039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:04.671218 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:04.672246 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:04.672251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:04.672419 (+ 168us) client_negotiation.cc:769] Sending connection context
0619 06:33:04.672553 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 06:33:04.672838 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 06:33:04.675292 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.652579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.0.97.124:44703)
0619 06:33:04.652725 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.652730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.652746 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:33:04.652796 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.652801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.652865 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.652962 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.655880 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.657156 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.658964 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.659747 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.659876 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.664208 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:04.664215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:04.664220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:04.664258 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:04.669589 (+ 5331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.670582 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.670590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.670594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.670840 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:04.671410 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:04.671415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:04.671419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:04.672048 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:04.673719 (+ 1671us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.673959 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.674348 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 06:33:04.681326 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c: Bootstrap starting.
I20260619 06:33:04.685251 22919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:04.645829 22926 master_runner.cc:186] Error getting master registration for 127.0.97.125:36795: OK, instance_id { permanent_uuid: "f94ef3da1168450783ca62324397947c" instance_seqno: 1781850783927058 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:33:04.686389 22926 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:33:04.691466 23025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a: Generated new TSK 0
I20260619 06:33:04.692482 23025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 06:33:04.692821 22919 log.cc:824] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:04.698617 23042 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e: acquiring CA information for follower catalog manager: success
I20260619 06:33:04.725486 23042 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5094ee875dc0494d826a491025a7335e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:33:04.729519 22919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c: No bootstrap required, opened a new log
I20260619 06:33:04.746277 22919 raft_consensus.cc:348] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
I20260619 06:33:04.746903 22919 raft_consensus.cc:729] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f94ef3da1168450783ca62324397947c, State: Initialized, Role: FOLLOWER
I20260619 06:33:04.747732 22919 consensus_queue.cc:260] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } }
I20260619 06:33:04.751250 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.751860 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:04.753341 22894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "416cc7b19cb74877b3c2f7a99f69464a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f94ef3da1168450783ca62324397947c"
I20260619 06:33:04.753803 22919 sys_catalog.cc:564] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:04.754170 22893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:04.757959 22893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [term 1 FOLLOWER]: Refusing update from remote peer 416cc7b19cb74877b3c2f7a99f69464a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:04.758991 23002 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:04.793192 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 416cc7b19cb74877b3c2f7a99f69464a. Latest consensus state: current_term: 1 leader_uuid: "416cc7b19cb74877b3c2f7a99f69464a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.793550 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:04.794050 23051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "416cc7b19cb74877b3c2f7a99f69464a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:04.794534 23051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:04.795017 22894 raft_consensus.cc:2396] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 416cc7b19cb74877b3c2f7a99f69464a for term 1 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
W20260619 06:33:04.837601 22997 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:04.826493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38440 (local address 127.0.97.124:44703)
0619 06:33:04.826695 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.826703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.835573 (+ 8870us) server_negotiation.cc:407] Connection header received
0619 06:33:04.836006 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.836013 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.836092 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.836170 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.837325 (+ 1155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38440: BlockingRecv error: recv got EOF from 127.0.0.1:38440 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260619 06:33:04.844193 22999 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:04.830015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.0.97.126:45487)
0619 06:33:04.830204 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.830209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.837794 (+ 7585us) server_negotiation.cc:407] Connection header received
0619 06:33:04.837851 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.837855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.837930 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.838021 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.843945 (+ 5924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47422: BlockingRecv error: recv got EOF from 127.0.0.1:47422 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 06:33:04.846920 23001 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:04.844960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.0.97.125:36795)
0619 06:33:04.845139 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.845145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.845416 (+ 271us) server_negotiation.cc:407] Connection header received
0619 06:33:04.845609 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.845614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.845717 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.845779 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:04.846678 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33264: BlockingRecv error: recv got EOF from 127.0.0.1:33264 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 06:33:04.847987 23067 catalog_manager.cc:1259] Loaded cluster ID: 4a8e1199f94b4818b0f91f1515a283f8
I20260619 06:33:04.848080 23067 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c: loading cluster ID for follower catalog manager: success
I20260619 06:33:04.851804 23067 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c: acquiring CA information for follower catalog manager: success
I20260619 06:33:04.854735 23067 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f94ef3da1168450783ca62324397947c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:33:04.894726 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.879804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33272 (local address 127.0.97.125:36795)
0619 06:33:04.879976 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.879984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.887017 (+ 7033us) server_negotiation.cc:407] Connection header received
0619 06:33:04.887331 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.887338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.887559 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.887729 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:04.888459 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.889674 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.891732 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.892442 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.892535 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.893251 (+ 716us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:04.893941 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:04.894052 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.894281 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.894370 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 06:33:04.943984 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.928397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38444 (local address 127.0.97.124:44703)
0619 06:33:04.928597 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.928604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.936108 (+ 7504us) server_negotiation.cc:407] Connection header received
0619 06:33:04.936560 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.936568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.936763 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.936890 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:04.937531 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.938639 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.941240 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.941862 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:04.941950 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:04.942646 (+ 696us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:04.943284 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:04.943397 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:04.943589 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 06:33:04.943684 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:33:05.002381 22997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.983651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.0.97.124:44703)
0619 06:33:04.983849 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.983855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.995204 (+ 11349us) server_negotiation.cc:407] Connection header received
0619 06:33:04.995687 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.995693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.995780 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.995843 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:04.996547 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:04.997718 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:05.000025 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:05.000724 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:05.000816 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:05.001504 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:05.001684 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:05.001757 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:05.001990 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 06:33:05.002071 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 06:33:05.010919 23001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.996023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33278 (local address 127.0.97.125:36795)
0619 06:33:04.996214 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.996220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:05.004593 (+ 8373us) server_negotiation.cc:407] Connection header received
0619 06:33:05.004859 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:05.004865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:05.004959 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:05.005021 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:05.005588 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:05.006636 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:05.008612 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:05.009368 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:05.009491 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:05.010098 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:05.010237 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:05.010295 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:05.010505 (+ 210us) server_negotiation.cc:299] Negotiation successful
0619 06:33:05.010581 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:33:05.013451 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:04.989501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47426 (local address 127.0.97.126:45487)
0619 06:33:04.989690 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:04.989696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:04.995416 (+ 5720us) server_negotiation.cc:407] Connection header received
0619 06:33:04.996923 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:04.996930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:04.997199 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:04.997338 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:04.999573 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:05.000700 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:05.006308 (+ 5608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:05.006956 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:05.007051 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:05.012173 (+ 5122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:05.012678 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:05.012760 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:05.013027 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 06:33:05.013150 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 06:33:05.018525 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22778
I20260619 06:33:05.019644 22840 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:05.213588 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "416cc7b19cb74877b3c2f7a99f69464a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "416cc7b19cb74877b3c2f7a99f69464a" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 45487 } } peers { permanent_uuid: "f94ef3da1168450783ca62324397947c" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 36795 } } peers { permanent_uuid: "5094ee875dc0494d826a491025a7335e" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 44703 } } }
I20260619 06:33:05.213999 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 416cc7b19cb74877b3c2f7a99f69464a [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:05.245121 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22778
I20260619 06:33:05.272855 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22852
I20260619 06:33:05.274320 22914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:05.391528 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22852
I20260619 06:33:05.407929 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 22926
I20260619 06:33:05.408934 22988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:05.514182 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 22926
2026-06-19T06:33:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3848 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 06:33:05 dist-test-slave-4t1f krb5kdc[23096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 06:33:05 dist-test-slave-4t1f krb5kdc[23096](info): set up 2 sockets
Jun 19 06:33:05 dist-test-slave-4t1f krb5kdc[23096](info): commencing operation
krb5kdc: starting...
W20260619 06:33:06.625721 389 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/test-user.keytab.
Jun 19 06:33:06 dist-test-slave-4t1f krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T06:33:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T06:33:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
Entry for principal kudu/127.0.97.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab.
I20260619 06:33:06.860533 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44275
--webserver_interface=127.0.97.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44275,127.0.97.125:44863,127.0.97.124:43361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 06:33:07.176339 23109 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:33:07.176636 23109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:07.176708 23109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:07.176789 23109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:07.183912 23109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:07.184001 23109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:07.184057 23109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:07.184125 23109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:07.184180 23109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:07.193591 23109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44275,127.0.97.125:44863,127.0.97.124:43361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.126.keytab
--principal=kudu/127.0.97.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.126:44275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:07.194751 23109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:07.195978 23109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 06:33:07.207261 23115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:07.207454 23117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:07.207726 23114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:07.208076 23109 server_base.cc:1034] running on GCE node
I20260619 06:33:07.208841 23109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:07.210112 23109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:07.211342 23109 hybrid_clock.cc:648] HybridClock initialized: now 1781850787211267 us; error 44 us; skew 500 ppm
I20260619 06:33:07.211625 23109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:07 dist-test-slave-4t1f krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:07.217935 23109 init.cc:377] Logged in from keytab as kudu/127.0.97.126@KRBTEST.COM (short username kudu)
I20260619 06:33:07.219494 23109 webserver.cc:460] Webserver started at http://127.0.97.126:34045/ using document root <none> and password file <none>
I20260619 06:33:07.220137 23109 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:07.220232 23109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:07.220520 23109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:07.223608 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data/instance:
uuid: "fb21da6a1bdd4b14af0f7520f534dc88"
format_stamp: "Formatted at 2026-06-19 06:33:07 on dist-test-slave-4t1f"
server_key: "b994dc7761d8bb3826b3fc61b6b5b946"
server_key_iv: "0bea225871b69c47836b2d6b92647b50"
server_key_version: "encryptionkey@0"
I20260619 06:33:07.224351 23109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/wal/instance:
uuid: "fb21da6a1bdd4b14af0f7520f534dc88"
format_stamp: "Formatted at 2026-06-19 06:33:07 on dist-test-slave-4t1f"
server_key: "b994dc7761d8bb3826b3fc61b6b5b946"
server_key_iv: "0bea225871b69c47836b2d6b92647b50"
server_key_version: "encryptionkey@0"
I20260619 06:33:07.229416 23109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260619 06:33:07.232542 23124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:07.233829 23109 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260619 06:33:07.234018 23109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/wal
uuid: "fb21da6a1bdd4b14af0f7520f534dc88"
format_stamp: "Formatted at 2026-06-19 06:33:07 on dist-test-slave-4t1f"
server_key: "b994dc7761d8bb3826b3fc61b6b5b946"
server_key_iv: "0bea225871b69c47836b2d6b92647b50"
server_key_version: "encryptionkey@0"
I20260619 06:33:07.234256 23109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:07.266819 23109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:07.270653 23109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:07.270932 23109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:07.289145 23109 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.126:44275
I20260619 06:33:07.289172 23175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.126:44275 every 8 connection(s)
I20260619 06:33:07.290467 23109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/data/info.pb
I20260619 06:33:07.293727 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 23109
I20260619 06:33:07.293936 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-0/wal/instance
I20260619 06:33:07.294077 23176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:07.294432 389 external_mini_cluster.cc:1428] Setting key 93bef65d4bf291120c99d64b9c9f936c
I20260619 06:33:07.300371 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:07 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.125@KRBTEST.COM" created.
I20260619 06:33:07.322258 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.303484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44275 (local address 127.0.0.1:53198)
0619 06:33:07.304336 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.304381 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:07.306337 (+ 1956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:07.307132 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:07.307138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:07.307821 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:07.308156 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.308172 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.309760 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.309767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.310817 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.310829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.311734 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.311741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.311909 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.312580 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:07.312600 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:07.314967 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:07.318129 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:07.318155 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:07.318167 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:07.318546 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:07.318937 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:07.318943 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:07.318949 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:07.319106 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:07.319831 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:07.319838 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:07.320056 (+ 218us) client_negotiation.cc:769] Sending connection context
0619 06:33:07.320294 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 06:33:07.320906 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":155,"thread_start_us":376,"threads_started":1}
I20260619 06:33:07.322292 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.303485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.0.97.126:44275)
0619 06:33:07.305148 (+ 1663us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:07.305161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:07.305207 (+ 46us) server_negotiation.cc:407] Connection header received
0619 06:33:07.306686 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:07.306692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:07.306907 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:07.307110 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:07.308362 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.309614 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.310957 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.311497 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.311700 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.315185 (+ 3485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:07.315193 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:07.315204 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:07.315230 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:07.318014 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:07.318670 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:07.318680 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:07.318685 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:07.318831 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:07.319209 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:07.319216 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:07.319220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:07.319696 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:07.319905 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:07.320405 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 06:33:07.320788 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":1435,"thread_start_us":1194,"threads_started":1}
I20260619 06:33:07.325628 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:33:07.326673 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.326264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41148)
0619 06:33:07.326457 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.326564 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 06:33:07.326956 23126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
I20260619 06:33:07.328339 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.328021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41160)
0619 06:33:07.328156 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.328223 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 06:33:07.328522 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab.
Entry for principal kudu/127.0.97.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab.
I20260619 06:33:07.348336 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:44863
--webserver_interface=127.0.97.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44275,127.0.97.125:44863,127.0.97.124:43361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:33:07.377912 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } attempt: 1
I20260619 06:33:07.379035 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.378607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41168)
0619 06:33:07.378799 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.378900 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:33:07.380016 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.379697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41184)
0619 06:33:07.379845 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.379902 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260619 06:33:07.380208 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
I20260619 06:33:07.448577 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } attempt: 2
I20260619 06:33:07.449811 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.449316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41200)
0619 06:33:07.449578 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.449665 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260619 06:33:07.450851 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.450512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41212)
0619 06:33:07.450662 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.450732 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:33:07.451032 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
I20260619 06:33:07.542295 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } attempt: 3
I20260619 06:33:07.543401 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.542967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41218)
0619 06:33:07.543168 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.543266 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 06:33:07.544468 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.544117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41232)
0619 06:33:07.544261 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.544336 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:33:07.544658 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
W20260619 06:33:07.660547 23183 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:33:07.660888 23183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:07.661015 23183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:07.661177 23183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:07.668059 23183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:07.668185 23183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:07.668298 23183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:07.668390 23183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:07.668478 23183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:07.677759 23183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44275,127.0.97.125:44863,127.0.97.124:43361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.125.keytab
--principal=kudu/127.0.97.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.125:44863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:07.679059 23183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:07.680303 23183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:33:07.687878 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } attempt: 4
I20260619 06:33:07.689044 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.688467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41244)
0619 06:33:07.688775 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.688854 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260619 06:33:07.690264 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.689785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41248)
0619 06:33:07.690016 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.690111 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260619 06:33:07.690599 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.125:44863: Network error: Client connection negotiation failed: client connection to 127.0.97.125:44863: connect: Connection refused (error 111)
W20260619 06:33:07.691800 23188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:07.691954 23191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:07.692291 23189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:07.693176 23183 server_base.cc:1034] running on GCE node
I20260619 06:33:07.693858 23183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:07.695122 23183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:07.696357 23183 hybrid_clock.cc:648] HybridClock initialized: now 1781850787696293 us; error 38 us; skew 500 ppm
I20260619 06:33:07.696668 23183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:07 dist-test-slave-4t1f krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:07.699882 23183 init.cc:377] Logged in from keytab as kudu/127.0.97.125@KRBTEST.COM (short username kudu)
I20260619 06:33:07.701295 23183 webserver.cc:460] Webserver started at http://127.0.97.125:46753/ using document root <none> and password file <none>
I20260619 06:33:07.701930 23183 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:07.702054 23183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:07.702433 23183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:07.705180 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data/instance:
uuid: "6c6bf3835642429cad6da68801054d8b"
format_stamp: "Formatted at 2026-06-19 06:33:07 on dist-test-slave-4t1f"
server_key: "7961a4224e9a8930802c7c877e3a255f"
server_key_iv: "b1ff34937b77665168c75604349c32d5"
server_key_version: "encryptionkey@0"
I20260619 06:33:07.705915 23183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/wal/instance:
uuid: "6c6bf3835642429cad6da68801054d8b"
format_stamp: "Formatted at 2026-06-19 06:33:07 on dist-test-slave-4t1f"
server_key: "7961a4224e9a8930802c7c877e3a255f"
server_key_iv: "b1ff34937b77665168c75604349c32d5"
server_key_version: "encryptionkey@0"
I20260619 06:33:07.710632 23183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260619 06:33:07.713786 23198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:07.715004 23183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:07.715179 23183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/wal
uuid: "6c6bf3835642429cad6da68801054d8b"
format_stamp: "Formatted at 2026-06-19 06:33:07 on dist-test-slave-4t1f"
server_key: "7961a4224e9a8930802c7c877e3a255f"
server_key_iv: "b1ff34937b77665168c75604349c32d5"
server_key_version: "encryptionkey@0"
I20260619 06:33:07.715378 23183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:07.730962 23183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:07.734999 23183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:07.735251 23183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:07.752724 23183 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.125:44863
I20260619 06:33:07.752732 23249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.125:44863 every 8 connection(s)
I20260619 06:33:07.754050 23183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/data/info.pb
I20260619 06:33:07.757493 23250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:07.760215 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 23183
I20260619 06:33:07.760393 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-1/wal/instance
I20260619 06:33:07.760896 389 external_mini_cluster.cc:1428] Setting key 534b8e0864b0a31aaa0656ad54100f75
I20260619 06:33:07.766024 23250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:07 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.124@KRBTEST.COM; defaulting to no policy
I20260619 06:33:07.788097 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.769333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53212 (local address 127.0.97.126:44275)
0619 06:33:07.769566 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:07.769575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:07.770616 (+ 1041us) server_negotiation.cc:407] Connection header received
0619 06:33:07.772409 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:07.772418 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:07.772510 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:07.772629 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:07.774070 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.775233 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.777548 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.778107 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.778253 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.782324 (+ 4071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:07.782329 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:07.782332 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:07.782363 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:07.785140 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:07.785832 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:07.785837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:07.785841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:07.785963 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:07.786376 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:07.786381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:07.786384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:07.786656 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:07.786821 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:07.787565 (+ 744us) server_negotiation.cc:299] Negotiation successful
0619 06:33:07.787783 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:33:07.789008 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.769666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44275 (local address 127.0.0.1:53212)
0619 06:33:07.770434 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.770489 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:07.772150 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:07.772794 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:07.772801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:07.773416 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:07.773907 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.773922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.775381 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.775384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.777385 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.777401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.778358 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.778368 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.778699 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.779396 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:07.779441 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:07.782093 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:07.785270 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:07.785279 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:07.785294 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:07.785706 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:07.786080 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:07.786086 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:07.786089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:07.786259 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:07.786782 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:07.786791 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:07.787192 (+ 401us) client_negotiation.cc:769] Sending connection context
0619 06:33:07.787455 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 06:33:07.787925 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":302,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.124@KRBTEST.COM" created.
I20260619 06:33:07.790793 23250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:07 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:33:07.806782 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.791607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41262)
0619 06:33:07.791837 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.791858 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:07.791962 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:07.792868 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:07.792876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:07.793389 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:07.793598 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.793612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.794963 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.794971 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.796492 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.796509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.797334 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.797343 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.797581 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.798179 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:07.798195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:07.800251 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:07.804578 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:07.804582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:07.804585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:07.804915 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:07.805337 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:07.805340 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:07.805342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:07.805470 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:07.806138 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:07.806144 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:07.806244 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 06:33:07.806345 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 06:33:07.806519 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 06:33:07.807436 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.791814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.0.97.125:44863)
0619 06:33:07.792474 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:07.792479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:07.792502 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:33:07.792542 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:07.792547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:07.792708 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:07.792817 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:07.793767 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.794810 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.796636 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.797193 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.797283 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.800465 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:07.800480 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:07.800498 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:07.800538 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:07.804455 (+ 3917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:07.805042 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:07.805053 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:07.805080 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:07.805229 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:07.805575 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:07.805582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:07.805588 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:07.806017 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:07.806224 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:07.806526 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 06:33:07.806789 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":380,"threads_started":1}
I20260619 06:33:07.809726 23250 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:33:07.810828 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.810424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60900)
0619 06:33:07.810592 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.810704 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 06:33:07.811115 23202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
I20260619 06:33:07.812570 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.812276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60908)
0619 06:33:07.812400 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.812470 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 06:33:07.812870 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab.
Entry for principal kudu/127.0.97.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab.
I20260619 06:33:07.821949 389 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskWesBqs/build/asan/bin/kudu
/tmp/dist-test-taskWesBqs/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:43361
--webserver_interface=127.0.97.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.84:35537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.126:44275,127.0.97.125:44863,127.0.97.124:43361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 06:33:07.862164 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 1
I20260619 06:33:07.863251 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.862802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60916)
0619 06:33:07.862992 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.863108 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260619 06:33:07.864189 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.863863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60930)
0619 06:33:07.864019 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.864087 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 06:33:07.864377 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
I20260619 06:33:07.932655 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 2
I20260619 06:33:07.933858 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.933330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60934)
0619 06:33:07.933554 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.933667 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 06:33:07.934888 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.934489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60938)
0619 06:33:07.934655 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.934737 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260619 06:33:07.935103 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
I20260619 06:33:07.989966 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } attempt: 5
Jun 19 06:33:07 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:33:08.003255 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.990832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41272 (local address 127.0.97.125:44863)
0619 06:33:07.991006 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:07.991013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:07.991054 (+ 41us) server_negotiation.cc:407] Connection header received
0619 06:33:07.991239 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:07.991246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:07.991327 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:07.991434 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:07.992245 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.992933 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.994126 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.994575 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.994713 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.997992 (+ 3279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:07.997997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:07.998000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:07.998038 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.000602 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.001331 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.001339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.001346 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.001462 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.001827 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.001831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.001836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.002114 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.002267 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.002687 (+ 420us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.002917 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 06:33:08.003664 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:07.990718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41272)
0619 06:33:07.990940 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:07.990961 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:07.991094 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:07.991470 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:07.991476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:07.991895 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:07.992061 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.992082 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.993110 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.993116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.993973 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:07.993983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:07.994726 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:07.994732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:07.994950 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:07.995589 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:07.995607 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:07.997800 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:08.000757 (+ 2957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.000764 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.000768 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.001210 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.001574 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.001579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.001582 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.001716 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.002232 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:08.002237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:08.002401 (+ 164us) client_negotiation.cc:769] Sending connection context
0619 06:33:08.002553 (+ 152us) client_negotiation.cc:240] Negotiation successful
0619 06:33:08.003345 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260619 06:33:08.004645 23176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } has no permanent_uuid. Determining permanent_uuid...
I20260619 06:33:08.005831 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.005477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60942)
0619 06:33:08.005639 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.005708 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 06:33:08.006808 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.006491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60950)
0619 06:33:08.006655 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.006721 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 06:33:08.007035 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
I20260619 06:33:08.026324 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 3
I20260619 06:33:08.027436 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.026963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60964)
0619 06:33:08.027184 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.027274 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260619 06:33:08.028522 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.028122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60970)
0619 06:33:08.028346 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.028414 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
W20260619 06:33:08.028815 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
I20260619 06:33:08.058259 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 1
I20260619 06:33:08.059577 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.058878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60984)
0619 06:33:08.059183 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.059323 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260619 06:33:08.060807 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.060316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:60996)
0619 06:33:08.060505 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.060624 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 06:33:08.061106 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
I20260619 06:33:08.129436 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 2
I20260619 06:33:08.130630 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.130182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32776)
0619 06:33:08.130389 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.130480 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 06:33:08.131614 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.131264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32792)
0619 06:33:08.131432 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.131496 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260619 06:33:08.131825 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
W20260619 06:33:08.148356 23257 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 06:33:08.148713 23257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 06:33:08.148829 23257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 06:33:08.148957 23257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 06:33:08.156142 23257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 06:33:08.156256 23257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 06:33:08.156386 23257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 06:33:08.156476 23257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 06:33:08.156584 23257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 06:33:08.166440 23257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.84:35537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.126:44275,127.0.97.125:44863,127.0.97.124:43361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/krb5kdc/kudu_127.0.97.124.keytab
--principal=kudu/127.0.97.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.124:43361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 19 Jun 2026 05:40:48 UTC on ccycloud.quasar-zrzaui.root.comops.site
build id 4455154
ASAN enabled
I20260619 06:33:08.167644 23257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 06:33:08.168897 23257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 06:33:08.172132 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 4
I20260619 06:33:08.173337 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.172729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32802)
0619 06:33:08.172956 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.173043 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 06:33:08.174477 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.174078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32816)
0619 06:33:08.174221 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.174331 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 06:33:08.174688 23250 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
W20260619 06:33:08.180610 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:08.180646 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 06:33:08.180675 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 06:33:08.181703 23257 server_base.cc:1034] running on GCE node
I20260619 06:33:08.182430 23257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 06:33:08.183707 23257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 06:33:08.184973 23257 hybrid_clock.cc:648] HybridClock initialized: now 1781850788184917 us; error 58 us; skew 500 ppm
I20260619 06:33:08.185312 23257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 06:33:08.188509 23257 init.cc:377] Logged in from keytab as kudu/127.0.97.124@KRBTEST.COM (short username kudu)
I20260619 06:33:08.189836 23257 webserver.cc:460] Webserver started at http://127.0.97.124:41393/ using document root <none> and password file <none>
I20260619 06:33:08.190481 23257 fs_manager.cc:359] Metadata directory not provided
I20260619 06:33:08.190587 23257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 06:33:08.190897 23257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 06:33:08.193715 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data/instance:
uuid: "3bda5da59bec4d3a85b3605a11af8ef9"
format_stamp: "Formatted at 2026-06-19 06:33:08 on dist-test-slave-4t1f"
server_key: "85e5b0747b23a53cfe34900fe4250d44"
server_key_iv: "6ba70926e849c53cdf083af5d8f77624"
server_key_version: "encryptionkey@0"
I20260619 06:33:08.194464 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/wal/instance:
uuid: "3bda5da59bec4d3a85b3605a11af8ef9"
format_stamp: "Formatted at 2026-06-19 06:33:08 on dist-test-slave-4t1f"
server_key: "85e5b0747b23a53cfe34900fe4250d44"
server_key_iv: "6ba70926e849c53cdf083af5d8f77624"
server_key_version: "encryptionkey@0"
I20260619 06:33:08.199102 23257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260619 06:33:08.202209 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 06:33:08.203464 23257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 06:33:08.203629 23257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data,/tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/wal
uuid: "3bda5da59bec4d3a85b3605a11af8ef9"
format_stamp: "Formatted at 2026-06-19 06:33:08 on dist-test-slave-4t1f"
server_key: "85e5b0747b23a53cfe34900fe4250d44"
server_key_iv: "6ba70926e849c53cdf083af5d8f77624"
server_key_version: "encryptionkey@0"
I20260619 06:33:08.203827 23257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 06:33:08.219877 23257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 06:33:08.223526 23257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 06:33:08.223785 23257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 06:33:08.238013 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 3
I20260619 06:33:08.239300 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.238734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32826)
0619 06:33:08.239072 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.239152 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260619 06:33:08.240535 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.240044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32832)
0619 06:33:08.240242 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.240361 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260619 06:33:08.240794 23176 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.124:43361: Network error: Client connection negotiation failed: client connection to 127.0.97.124:43361: connect: Connection refused (error 111)
I20260619 06:33:08.242769 23257 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.124:43361
I20260619 06:33:08.242805 23323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.124:43361 every 8 connection(s)
I20260619 06:33:08.243999 23257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/data/info.pb
I20260619 06:33:08.245136 389 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskWesBqs/build/asan/bin/kudu as pid 23257
I20260619 06:33:08.245313 389 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskWesBqs/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781850433974487-389-0/minicluster-data/master-2/wal/instance
I20260619 06:33:08.245817 389 external_mini_cluster.cc:1428] Setting key afcf9a5e51098f16d41eba25ce0f276e
I20260619 06:33:08.248240 23324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 06:33:08.256086 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
I20260619 06:33:08.268097 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.247414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.0.97.124:43361)
0619 06:33:08.248179 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.248189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.248236 (+ 47us) server_negotiation.cc:407] Connection header received
0619 06:33:08.249853 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.249872 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.250216 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.250510 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.252136 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.253466 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.255355 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.255994 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.256140 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.260995 (+ 4855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.261002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.261011 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.261040 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.264156 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.264828 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.264836 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.264843 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.265005 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.265436 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.265442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.265448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.265907 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.266143 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.266560 (+ 417us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.266968 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":374,"threads_started":1}
I20260619 06:33:08.274159 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.258932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.126:44275 (local address 127.0.0.1:53216)
0619 06:33:08.259713 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.259753 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:08.259947 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:08.260333 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:08.260340 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:08.261154 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:08.261442 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.261450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.262857 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.262860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.263962 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.263976 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.264789 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.264796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.264930 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.265552 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:08.265569 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:08.267942 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:08.271733 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.271742 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.271746 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.272324 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.272677 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.272682 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.272686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.272844 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.273376 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:08.273389 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:08.273550 (+ 161us) client_negotiation.cc:769] Sending connection context
0619 06:33:08.273687 (+ 137us) client_negotiation.cc:240] Negotiation successful
0619 06:33:08.273883 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":393,"threads_started":1}
I20260619 06:33:08.274384 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.258943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53216 (local address 127.0.97.126:44275)
0619 06:33:08.259164 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.259171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.259831 (+ 660us) server_negotiation.cc:407] Connection header received
0619 06:33:08.260086 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.260092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.260152 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.260222 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.261595 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.262669 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.264104 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.264607 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.264696 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.268134 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.268141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.268145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.268186 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.271623 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.272437 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.272440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.272443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.272553 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.272948 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.272951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.272953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.273250 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.273401 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.273815 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.274021 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:33:08.275678 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.126@KRBTEST.COM
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
I20260619 06:33:08.286376 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.271006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.0.97.126:44275)
0619 06:33:08.271587 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.271593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.271616 (+ 23us) server_negotiation.cc:407] Connection header received
0619 06:33:08.271659 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.271667 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.271739 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.271825 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.273380 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.274511 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.275869 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.276480 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.276663 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.281114 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.281124 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.281130 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.281181 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.283867 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.284462 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.284468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.284476 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.284589 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.284923 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.284928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.284931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.285267 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.285522 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.285813 (+ 291us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.286012 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":240,"threads_started":1}
I20260619 06:33:08.289561 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.276294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.125:44863 (local address 127.0.0.1:41284)
0619 06:33:08.276440 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.276459 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:08.276561 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:08.276985 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:08.276991 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:08.277461 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:08.277663 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.277674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.278605 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.278612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.279854 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.279866 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.280826 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.280830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.280997 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.281496 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:08.281516 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:08.283730 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:08.286908 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.286915 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.286918 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.287299 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.287706 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.287713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.287716 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.287872 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.288477 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:08.288481 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:08.288755 (+ 274us) client_negotiation.cc:769] Sending connection context
0619 06:33:08.288889 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 06:33:08.289182 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260619 06:33:08.290453 23254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.276475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.0.97.125:44863)
0619 06:33:08.276690 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.276697 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.276716 (+ 19us) server_negotiation.cc:407] Connection header received
0619 06:33:08.276775 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.276781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.276851 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.276925 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.277814 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.278465 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.280016 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.280687 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.280822 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.283907 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.283912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.283915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.283945 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.286790 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.287422 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.287427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.287430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.287589 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.287964 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.287969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.287972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.288331 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.288514 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.289857 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.290100 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 06:33:08.291344 23324 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.125@KRBTEST.COM
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.124@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:33:08.305454 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.292124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32858)
0619 06:33:08.292680 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.292697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:08.292814 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:08.293248 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:08.293254 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:08.293690 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:08.293855 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.293866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.294860 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.294863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.296127 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.296141 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.297259 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.297265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.297467 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.298023 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:08.298042 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:08.300125 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:08.302954 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.302957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.302960 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.303414 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.303794 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.303797 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.303799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.303911 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.304408 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:08.304412 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:08.304575 (+ 163us) client_negotiation.cc:769] Sending connection context
0619 06:33:08.304759 (+ 184us) client_negotiation.cc:240] Negotiation successful
0619 06:33:08.305018 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447}
I20260619 06:33:08.305498 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.292050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.0.97.124:43361)
0619 06:33:08.292214 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.292219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.292747 (+ 528us) server_negotiation.cc:407] Connection header received
0619 06:33:08.292918 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.292923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.292979 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.293034 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.293993 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.294751 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.296270 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.296954 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.297050 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.300284 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.300288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.300291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.300317 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.302826 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.303515 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.303519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.303526 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.303644 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.304031 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.304035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.304037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.304260 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.304398 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.304873 (+ 475us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.305248 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":60}
I20260619 06:33:08.305943 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.287972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.97.125:44863)
0619 06:33:08.288552 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.288557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.288752 (+ 195us) server_negotiation.cc:407] Connection header received
0619 06:33:08.288876 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.288881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.288943 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.289044 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.290719 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.292434 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.294228 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.294884 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.294999 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.299855 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.299861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.299866 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.299902 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.302709 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.303488 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.303495 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.303498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.303634 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.304008 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.304013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.304018 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.304270 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.304562 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.304809 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.305016 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":211,"threads_started":1}
I20260619 06:33:08.310546 389 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 06:33:08.311393 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: Bootstrap starting.
I20260619 06:33:08.314843 23324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:08.315786 23324 log.cc:824] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:08.313556 23151 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fb21da6a1bdd4b14af0f7520f534dc88"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53218
I20260619 06:33:08.318364 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: No bootstrap required, opened a new log
I20260619 06:33:08.325325 23324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
I20260619 06:33:08.325624 23324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bda5da59bec4d3a85b3605a11af8ef9, State: Initialized, Role: FOLLOWER
I20260619 06:33:08.326040 23324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
I20260619 06:33:08.326902 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.327275 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:08.327745 23324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 06:33:08.336752 23343 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:33:08.336875 23343 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:33:08.334424 23257 master_runner.cc:186] Error getting master registration for 127.0.97.126:44275: OK, instance_id { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" instance_seqno: 1781850787283975 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:33:08.345448 23257 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 06:33:08.418020 23176 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 4
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.126@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:33:08.433333 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.418787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:32874)
0619 06:33:08.418996 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.419018 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:08.419163 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:08.419510 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:08.419515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:08.419999 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:08.420169 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.420177 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.421491 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.421496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.422417 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.422429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.423171 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.423177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.423362 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.424001 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:08.424036 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:08.426459 (+ 2423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:08.430610 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.430615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.430620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.431121 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.431484 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.431488 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.431492 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.431655 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.432309 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:08.432314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:08.432479 (+ 165us) client_negotiation.cc:769] Sending connection context
0619 06:33:08.432626 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 06:33:08.432893 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 06:33:08.433367 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.418864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32874 (local address 127.0.97.124:43361)
0619 06:33:08.419051 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.419059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.419085 (+ 26us) server_negotiation.cc:407] Connection header received
0619 06:33:08.419305 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.419309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.419372 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.419478 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.420315 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.421357 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.422562 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.423004 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.423156 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.426631 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.426638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.426643 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.426686 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.430463 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.431230 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.431236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.431240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.431376 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.431762 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.431767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.431770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.432189 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.432357 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.432740 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.432948 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:33:08.439175 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88: Bootstrap starting.
I20260619 06:33:08.443087 23176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:08.444299 23176 log.cc:824] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:08.447238 23176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88: No bootstrap required, opened a new log
I20260619 06:33:08.452154 23176 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
I20260619 06:33:08.452507 23176 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb21da6a1bdd4b14af0f7520f534dc88, State: Initialized, Role: FOLLOWER
I20260619 06:33:08.453032 23176 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
I20260619 06:33:08.459136 23346 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.459540 23346 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:08.460278 23176 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:08.460341 23151 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 06:33:08.460522 23151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:08.464356 23151 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
W20260619 06:33:08.470172 23357 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 06:33:08.470340 23357 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 06:33:08.470687 23151 leader_election.cc:288] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [CANDIDATE]: Term 1 election: Requested vote from peers 6c6bf3835642429cad6da68801054d8b (127.0.97.125:44863), 3bda5da59bec4d3a85b3605a11af8ef9 (127.0.97.124:43361)
I20260619 06:33:08.471954 23299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3bda5da59bec4d3a85b3605a11af8ef9"
I20260619 06:33:08.472419 23299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:08.473872 23250 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } attempt: 5
W20260619 06:33:08.475788 23278 tablet.cc:2369] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:08.475948 23299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb21da6a1bdd4b14af0f7520f534dc88 in term 1.
I20260619 06:33:08.476584 23128 leader_election.cc:302] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bda5da59bec4d3a85b3605a11af8ef9, fb21da6a1bdd4b14af0f7520f534dc88; no voters:
I20260619 06:33:08.477000 23346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 06:33:08.469903 23109 master_runner.cc:186] Error getting master registration for 127.0.97.125:44863: OK, instance_id { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" instance_seqno: 1781850787747743 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 06:33:08.482090 23109 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 19 06:33:08 dist-test-slave-4t1f krb5kdc[23096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781850787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.125@KRBTEST.COM for kudu/127.0.97.124@KRBTEST.COM
I20260619 06:33:08.484897 23346 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [term 1 LEADER]: Becoming Leader. State: Replica: fb21da6a1bdd4b14af0f7520f534dc88, State: Running, Role: LEADER
I20260619 06:33:08.485450 23346 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
I20260619 06:33:08.506712 23252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.474364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.124:43361 (local address 127.0.0.1:53276)
0619 06:33:08.474624 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 06:33:08.474643 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 06:33:08.474751 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 06:33:08.475306 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 06:33:08.475312 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 06:33:08.475852 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0619 06:33:08.476072 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.476089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.479005 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.479013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.480475 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 06:33:08.480490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.481738 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.481744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 06:33:08.481950 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.482632 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 06:33:08.482663 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 06:33:08.491186 (+ 8523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 06:33:08.499837 (+ 8651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.499844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.499851 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.500540 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.503166 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 06:33:08.503175 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 06:33:08.503180 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 06:33:08.503403 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 06:33:08.504977 (+ 1574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 06:33:08.504984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 06:33:08.505551 (+ 567us) client_negotiation.cc:769] Sending connection context
0619 06:33:08.505775 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 06:33:08.506162 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260619 06:33:08.509845 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.474567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.0.97.124:43361)
0619 06:33:08.474738 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.474744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.474762 (+ 18us) server_negotiation.cc:407] Connection header received
0619 06:33:08.474903 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.474908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.475002 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.475108 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.477475 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.478803 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.480664 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.481426 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.481543 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.491567 (+ 10024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.491575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.491579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.491624 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.499311 (+ 7687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.502762 (+ 3451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.502770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.502774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.502972 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.504274 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.504279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.504283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.504732 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.504892 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.508294 (+ 3402us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.509346 (+ 1052us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 06:33:08.516781 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b: Bootstrap starting.
W20260619 06:33:08.523581 23130 tablet.cc:2369] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 06:33:08.530164 23250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b: Neither blocks nor log segments found. Creating new log.
I20260619 06:33:08.531701 23250 log.cc:824] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b: Log is configured to *not* fsync() on all Append() calls
I20260619 06:33:08.536100 23360 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb21da6a1bdd4b14af0f7520f534dc88. Latest consensus state: current_term: 1 leader_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.536538 23360 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:08.547422 23250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b: No bootstrap required, opened a new log
I20260619 06:33:08.552814 23370 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 06:33:08.555145 23370 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 06:33:08.559242 23250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
I20260619 06:33:08.559768 23250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c6bf3835642429cad6da68801054d8b, State: Initialized, Role: FOLLOWER
I20260619 06:33:08.560348 23250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } }
I20260619 06:33:08.608892 23329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.486300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39296 (local address 127.0.97.126:44275)
0619 06:33:08.486509 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.486514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.537145 (+ 50631us) server_negotiation.cc:407] Connection header received
0619 06:33:08.538170 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.538179 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.538317 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.538423 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.553765 (+ 15342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.554927 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.563496 (+ 8569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.564446 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.564573 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.585261 (+ 20688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.585272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.585280 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.585336 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.589900 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.596945 (+ 7045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.596955 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.596961 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.597167 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.598550 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.598559 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.598565 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.599052 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.599361 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.608071 (+ 8710us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.608403 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 06:33:08.610198 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.488151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.0.97.125:44863)
0619 06:33:08.488386 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.488395 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.539740 (+ 51345us) server_negotiation.cc:407] Connection header received
0619 06:33:08.548422 (+ 8682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.548427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.548501 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.548568 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.556859 (+ 8291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.558057 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.565179 (+ 7122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.566156 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.566278 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.590436 (+ 24158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.590447 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.590454 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.590508 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.595194 (+ 4686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.597747 (+ 2553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.597756 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.597762 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.597934 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.599926 (+ 1992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.599934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.599940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.600425 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.600721 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.609420 (+ 8699us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.609713 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 06:33:08.615806 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:08.492804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.0.97.124:43361)
0619 06:33:08.524754 (+ 31950us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.524763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.551274 (+ 26511us) server_negotiation.cc:407] Connection header received
0619 06:33:08.551519 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.551525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.551608 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.551673 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.559923 (+ 8250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.560983 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.575408 (+ 14425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:08.576505 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:08.576675 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:08.601360 (+ 24685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 06:33:08.601370 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 06:33:08.601378 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 06:33:08.601449 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0619 06:33:08.606779 (+ 5330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.612584 (+ 5805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.612593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.612599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.612793 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 06:33:08.613356 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 06:33:08.613364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 06:33:08.613370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 06:33:08.613857 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 06:33:08.614159 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:08.614821 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 06:33:08.615152 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31747,"thread_start_us":42452,"threads_started":1}
I20260619 06:33:08.621196 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.621668 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:08.622609 23250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 06:33:08.622695 23225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6c6bf3835642429cad6da68801054d8b"
I20260619 06:33:08.623174 23225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [term 0 FOLLOWER]: Advancing to term 1
I20260619 06:33:08.628901 23299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [term 1 FOLLOWER]: Refusing update from remote peer fb21da6a1bdd4b14af0f7520f534dc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:08.630975 23360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:08.639484 23225 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb21da6a1bdd4b14af0f7520f534dc88 in term 1.
I20260619 06:33:08.640347 23224 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [term 1 FOLLOWER]: Refusing update from remote peer fb21da6a1bdd4b14af0f7520f534dc88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 06:33:08.642215 23360 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 06:33:08.669198 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb21da6a1bdd4b14af0f7520f534dc88. Latest consensus state: current_term: 1 leader_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.669528 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:08.699786 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.700150 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:08.705099 23360 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.705371 23360 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:08.706422 23370 catalog_manager.cc:1347] Generated new cluster ID: 96ad49e1a7a847b785a9a450e87a2611
I20260619 06:33:08.706533 23370 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 06:33:08.710142 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb21da6a1bdd4b14af0f7520f534dc88. Latest consensus state: current_term: 1 leader_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.710561 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:08.719370 23346 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.720153 23375 mvcc.cc:205] Tried to move back new op lower bound from 7298460830199017472 to 7298460829641547776. Current Snapshot: MvccSnapshot[applied={T|T < 7298460830199017472}]
I20260619 06:33:08.721112 23346 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88 [sys.catalog]: This master's current role is: LEADER
I20260619 06:33:08.732295 23370 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 06:33:08.735867 23370 catalog_manager.cc:1504] Loading token signing keys...
I20260619 06:33:08.736269 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb21da6a1bdd4b14af0f7520f534dc88" member_type: VOTER last_known_addr { host: "127.0.97.126" port: 44275 } } peers { permanent_uuid: "6c6bf3835642429cad6da68801054d8b" member_type: VOTER last_known_addr { host: "127.0.97.125" port: 44863 } } peers { permanent_uuid: "3bda5da59bec4d3a85b3605a11af8ef9" member_type: VOTER last_known_addr { host: "127.0.97.124" port: 43361 } } }
I20260619 06:33:08.736501 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 06:33:08.773334 23370 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb21da6a1bdd4b14af0f7520f534dc88: Generated new TSK 0
I20260619 06:33:08.773406 23392 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 06:33:08.774938 23370 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 06:33:08.838822 23329 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.835371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39302 (local address 127.0.97.126:44275)
0619 06:33:08.835575 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.835584 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.836301 (+ 717us) server_negotiation.cc:407] Connection header received
0619 06:33:08.836624 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.836640 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.836784 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.836874 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.838396 (+ 1522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39302: BlockingRecv error: recv got EOF from 127.0.0.1:39302 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260619 06:33:08.841460 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.836111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.0.97.125:44863)
0619 06:33:08.836289 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.836295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.839464 (+ 3169us) server_negotiation.cc:407] Connection header received
0619 06:33:08.839735 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.839744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.839861 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.839989 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.841143 (+ 1154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47420: BlockingRecv error: recv got EOF from 127.0.0.1:47420 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 06:33:08.845176 23327 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.842792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53298 (local address 127.0.97.124:43361)
0619 06:33:08.842981 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.842987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.843112 (+ 125us) server_negotiation.cc:407] Connection header received
0619 06:33:08.843408 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.843417 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.843524 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.843609 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.844781 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53298: BlockingRecv error: recv got EOF from 127.0.0.1:53298 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 06:33:08.858810 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.852345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.0.97.125:44863)
0619 06:33:08.852546 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.852553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.857522 (+ 4969us) server_negotiation.cc:407] Connection header received
0619 06:33:08.857749 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.857753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.857853 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.857961 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.858656 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47424: BlockingRecv error: recv got EOF from 127.0.0.1:47424 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:33:08.873854 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.871878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47430 (local address 127.0.97.125:44863)
0619 06:33:08.872110 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.872117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.872589 (+ 472us) server_negotiation.cc:407] Connection header received
0619 06:33:08.872779 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.872784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.872854 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.872955 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.873611 (+ 656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47430: BlockingRecv error: recv got EOF from 127.0.0.1:47430 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 06:33:08.880816 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.878680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.0.97.125:44863)
0619 06:33:08.878873 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.878881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.879354 (+ 473us) server_negotiation.cc:407] Connection header received
0619 06:33:08.879544 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.879549 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.879652 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.879766 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.880578 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47440: BlockingRecv error: recv got EOF from 127.0.0.1:47440 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 06:33:08.889564 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.887533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.0.97.125:44863)
0619 06:33:08.887791 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.887799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.888292 (+ 493us) server_negotiation.cc:407] Connection header received
0619 06:33:08.888496 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.888502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.888578 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.888682 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.889332 (+ 650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47442: BlockingRecv error: recv got EOF from 127.0.0.1:47442 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260619 06:33:08.903052 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.900355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.0.97.125:44863)
0619 06:33:08.900548 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.900557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.901040 (+ 483us) server_negotiation.cc:407] Connection header received
0619 06:33:08.901271 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.901276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.901360 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.901472 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.902869 (+ 1397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47458: BlockingRecv error: recv got EOF from 127.0.0.1:47458 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 06:33:08.924664 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.922544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.0.97.125:44863)
0619 06:33:08.922752 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.922761 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.923308 (+ 547us) server_negotiation.cc:407] Connection header received
0619 06:33:08.923508 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.923522 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.923586 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.923688 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.924397 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47468: BlockingRecv error: recv got EOF from 127.0.0.1:47468 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 06:33:08.961710 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:08.959737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.0.97.125:44863)
0619 06:33:08.959959 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:08.959966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:08.960226 (+ 260us) server_negotiation.cc:407] Connection header received
0619 06:33:08.960456 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:08.960462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:08.960558 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:08.960683 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:08.961463 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47470: BlockingRecv error: recv got EOF from 127.0.0.1:47470 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260619 06:33:09.031596 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:09.029351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47476 (local address 127.0.97.125:44863)
0619 06:33:09.029624 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.029631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.030025 (+ 394us) server_negotiation.cc:407] Connection header received
0619 06:33:09.030293 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.030297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.030393 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.030484 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:09.031372 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47476: BlockingRecv error: recv got EOF from 127.0.0.1:47476 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260619 06:33:09.165278 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:09.163333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47478 (local address 127.0.97.125:44863)
0619 06:33:09.163596 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.163604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.163896 (+ 292us) server_negotiation.cc:407] Connection header received
0619 06:33:09.164107 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.164113 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.164197 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.164305 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:09.165084 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47478: BlockingRecv error: recv got EOF from 127.0.0.1:47478 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 06:33:09.338877 23343 catalog_manager.cc:1259] Loaded cluster ID: 96ad49e1a7a847b785a9a450e87a2611
I20260619 06:33:09.338979 23343 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: loading cluster ID for follower catalog manager: success
I20260619 06:33:09.341483 23343 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: acquiring CA information for follower catalog manager: success
I20260619 06:33:09.343112 23343 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3bda5da59bec4d3a85b3605a11af8ef9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 06:33:09.426848 23330 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 06:33:09.425305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47486 (local address 127.0.97.125:44863)
0619 06:33:09.425482 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.425486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.425687 (+ 201us) server_negotiation.cc:407] Connection header received
0619 06:33:09.425834 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.425839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.425919 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.426013 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 06:33:09.426636 (+ 623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47486: BlockingRecv error: recv got EOF from 127.0.0.1:47486 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 06:33:09.777462 23392 catalog_manager.cc:1259] Loaded cluster ID: 96ad49e1a7a847b785a9a450e87a2611
I20260619 06:33:09.777639 23392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b: loading cluster ID for follower catalog manager: success
I20260619 06:33:09.780153 23392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b: acquiring CA information for follower catalog manager: success
I20260619 06:33:09.781949 23392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c6bf3835642429cad6da68801054d8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 06:33:09.949563 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:09.942618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47490 (local address 127.0.97.125:44863)
0619 06:33:09.943423 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.943428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.943444 (+ 16us) server_negotiation.cc:407] Connection header received
0619 06:33:09.943505 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.943509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.943683 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.943788 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:09.944314 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.945173 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.947077 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.947577 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.947671 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:09.948237 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:09.948790 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:09.948898 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:09.949049 (+ 151us) server_negotiation.cc:299] Negotiation successful
0619 06:33:09.949165 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":534,"threads_started":1}
I20260619 06:33:09.966154 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:09.955909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.0.97.124:43361)
0619 06:33:09.960433 (+ 4524us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.960437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.960452 (+ 15us) server_negotiation.cc:407] Connection header received
0619 06:33:09.960549 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.960553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.960681 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.960805 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:09.961272 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.961992 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.963994 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.964429 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.964541 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:09.965038 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:09.965543 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:09.965653 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:09.965783 (+ 130us) server_negotiation.cc:299] Negotiation successful
0619 06:33:09.965844 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4406,"thread_start_us":4287,"threads_started":1}
I20260619 06:33:09.991804 23457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:09.972276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47496 (local address 127.0.97.125:44863)
0619 06:33:09.972443 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.972449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.984301 (+ 11852us) server_negotiation.cc:407] Connection header received
0619 06:33:09.984593 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.984602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.984723 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.984800 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:09.985656 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.986858 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.989068 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.989865 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.989971 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:09.990972 (+ 1001us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:09.991107 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:09.991168 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:09.991381 (+ 213us) server_negotiation.cc:299] Negotiation successful
0619 06:33:09.991464 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 06:33:09.993309 23463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:09.972082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.0.97.124:43361)
0619 06:33:09.972222 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.972228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.986783 (+ 14555us) server_negotiation.cc:407] Connection header received
0619 06:33:09.986924 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.986929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.987026 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.987134 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:09.987776 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.988855 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.991057 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.991679 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.991780 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:09.992513 (+ 733us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:09.992638 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:09.992705 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:09.992925 (+ 220us) server_negotiation.cc:299] Negotiation successful
0619 06:33:09.993015 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260619 06:33:09.994107 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 06:33:09.972270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39316 (local address 127.0.97.126:44275)
0619 06:33:09.985456 (+ 13186us) server_negotiation.cc:206] Beginning negotiation
0619 06:33:09.985464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 06:33:09.985485 (+ 21us) server_negotiation.cc:407] Connection header received
0619 06:33:09.985627 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 06:33:09.985634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 06:33:09.985865 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 06:33:09.986041 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 06:33:09.987357 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.988466 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.990825 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 06:33:09.991499 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 06:33:09.991622 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 06:33:09.992577 (+ 955us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 06:33:09.993002 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 06:33:09.993146 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 06:33:09.993692 (+ 546us) server_negotiation.cc:299] Negotiation successful
0619 06:33:09.993782 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13052,"thread_start_us":13876,"threads_started":1}
I20260619 06:33:10.000870 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 23109
I20260619 06:33:10.001978 23171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:10.134490 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 23109
I20260619 06:33:10.157929 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 23183
I20260619 06:33:10.159274 23245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:10.271163 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 23183
I20260619 06:33:10.290450 389 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskWesBqs/build/asan/bin/kudu pid 23257
I20260619 06:33:10.291476 23319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260619 06:33:10.397178 389 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskWesBqs/build/asan/bin/kudu with pid 23257
2026-06-19T06:33:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4878 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8726 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (356388 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST